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A Comparison of Alternative Models for Estimating School Performance in Mathematics 
and Reading/Language Arts in Four State Accountability Systems: Oregon Results 


Background and Introduction 

This technical report is one of a series of four technical reports that describe the results of 
a study comparing eight alternative models for estimating school academic achievement using 
data from the Arizona, North Carolina, Oregon, and Pennsylvania accountability systems. Our 
purpose was not to evaluate or examine the accountability systems in use by these states, but to 
evaluate a broader range of models commonly used for estimating school performance that are 
applied in many states and frequently reported in the school effectiveness research literature. 
This introduction briefly describes the study background and details the methods and procedures 
we used to estimate the eight school performance models and compare model results in all four 
states. The individual state technical reports including details on each state’s accountability data, 
assessment instruments, and results are provided at: http://www.ncaase.com/publications/tech- 
reports. 

Despite the central importance of analytic models used in evaluating teacher and school 
effects in modern accountability systems, there are relatively few studies of the reliability and 
validity of these high-stakes systems (see, for example, Goldschmidt, Choi, & Beaudoin, 2012). 
The results reported here examine eight models using operational state accountability data in 
mathematics and reading/language arts from the four participating states. We addressed four 
questions surrounding the use of analytic models for the evaluation of school performance: 

1. Are estimates of school performance stable across successive cohorts of students? 

2. How well do estimates of school performance correlate among models? 

3. How do estimates of school performance correlate with variables describing the 
student composition of the school? 

4. Do estimates of school performance vary from one model to another based on the 
school composition of students with disabilities (SWD)? 


General Method Description 

Sample 

The sample from each state is described in each individual state technical report. In three 
of the four states, the sample consisted of all students who took the state’s mathematics or 
reading/language arts general assessment in any one school year from 2007-08 through 2011-12, 
and whose records in each year were included in the state’s calculation of Adequate Yearly 
Progress (AYP). Samples were separated into two grade level bands: a longitudinal elementary 
school sample (Grades 3 through 5) and a longitudinal middle school sample (Grades 6 through 
8), each consisting of three cohorts (a) 2007/08 through 2009/2010; (b) 2008/09 through 
2010/11; and (c) 2009/10 through 2011/12 (see research design schematic below). In Arizona, 
only one elementary and middle school cohort was used (2006/07 through 2008/09) due to 
changes in the Arizona testing program in 2010. 


Instruments 

The outcome measures for all analyses were the standardized mathematics and 
reading/language arts tests used for accountability in each state. In three of the states, the 
instruments used vertically linked developmental scales created using item response theory (IRT) 
methods. In Pennsylvania, the test was not vertically linked over grades preventing the 
estimation of certain school performance models described in the next section. More detail 
about the Oregon test is contained in a later section of this report. 


Research design indicating academic years and longitudinal cohorts studied: 


Academic Year 


Grade 2007/08 2008/09 2009/10 2010/11 2011/12 
3 

4 

5 

6 

7 

8 N 


Note. E denotes an elementary school cohort, M denotes a middle school cohort. Only one 
elementary and one middle school cohort were available in the Arizona data. 


School Performance Models 

For all models, we estimated school performance in the last focal year (Grade 5 or 8) of 
the two grade level bands, as well as using prior years of achievement data as dictated by the 
particular model. We applied eight alternative analytic models of school performance to the 
mathematics and reading/language arts achievement data in elementary and middle school for 
each state. The eight school performance models were: Percent Proficient (PP), gain score 
(Gain), transition matrix (TM), student growth percentile (SGP), value-added model (VAM), and 
three Multilevel Linear Model (MLM) estimates: focal year intercept or status (MLMO), focal 
year growth rate (Grate), and average MLM growth rate across the three years (AvGrate). 


Percent Proficient (PP). PP was the NCLB required metric used by the state that 
calculated the percentage of students in each school that met or exceeded state benchmarks for 
proficiency in either mathematics or reading/language arts in each grade. 


Average Gain Score. Gain scores were calculated as the prior academic year (Grade 4 or 
Grade 7) scale score in mathematics or reading/language arts subtracted from the focal year scale 
score (Grade 5 or Grade 8): 


Gaini= Ai= Yit— Yit-1) (1) 


where Yit was the assessment outcome for student i at time ¢. Student gain scores were averaged 
for each school (labeled “Gain” below). 


Transition Matrix (TM). School performance estimates were computed from a table of 
the state’s proficiency categories in the prior year crossed with the proficiency categories in the 
focal year (Grade 5 or Grade 8) which, in the case of five proficiency categories, created a 
transition matrix table of 25 cells. The percentage of students occurring in each of the cells was 


Z 


entered and then a weighting scheme was applied to each cell and the products were summed to 
create a TM school performance index. The weighting scheme awarded one of three scores: (a) 
-1 was recorded if the student moved down one or more categories from the previous year, (b) 0 
was recorded if the student stayed in the same category, and (c) +1 was recorded if the student 
moved up one or more categories from the previous year (see Tindal, Nese, & Stevens, 2017). 
The weighted values were averaged across all cells to create an overall school TM index. 


Student Growth Percentiles (SGP). Student growth percentiles were computed at the 
student level using the approach described by Betebenner (2009). A student’s SGP was 
calculated by taking the current year test score and regressing it on the two prior years of test 
scores. Betebenner’s (2009) approach uses ordinal methods (quantile regression) as well as B- 
spline, cubic polynomial smoothing of the resulting normative distribution of conditional 
regression estimates. The analysis results in a relative rank for each student in a conditional 
distribution of those who had similar scores in previous years. We used the R package SGP 
(Betebenner, & Iwaarden, 2011) to compute student estimates based on the regression of the two 
prior years of test scores on the current year’s test score and then we aggregated student SGP for 
each school to create a median SGP as each school’s SGP performance estimate. 


Value-added Models (VAM). This mixed effects approach examined performance gains 
over years and included indicators for student membership in a particular school. This model is 
known generally as the “layered model” because layers of equations are added with each year of 
schooling (Ballou, Sanders, and Wright, 2004). For example, the model for our case with 
students with three years of data would be specified as follows: 


Yoij _ bo + Uo + €o (2a) 
Nij = by + Uo + Uy + ey (2b) 
Yoij = b> + Ug + Uy + U2 + C2, (2c) 


where Y,;; represents an assessment for student i at time ¢ (grade) attending school j. The fixed 
mean for all students in the combination of grades and schools was sui;, while ex; was the random 
deviation for student n from the mean, ju. The layered model we used was limited to a 
maximum of three years and was applied separately to mathematics and reading/language arts. 


Multilevel Linear Growth Model Initial Status, Focal Year Growth, and Average 
Growth (MLM0, MLM Growth Rate and MLM Average Growth Rate). We modeled student 
growth over the three elementary or three middle school grades with multilevel longitudinal 
analyses (Raudenbush & Bryk, 2002) using HLM 7.1 (Raudenbush, Bryk, Cheong, Congdon, & 
du Toit, 2011) and full maximum likelihood estimation. The conditional models included a 
level-1 model that specified student mathematics or reading/language arts scores predicted by a 
quadratic function of time of measurement, a level-2 model composed of the prediction of level- 
1 model parameters as a function of student mean values, and a level-3 model composed of the 
prediction of level-2 parameters as a function of school mean parameter values. Time was 
centered on the focal year (Grade 5 or 8) for computation of MLMO and MLM growth rate but 
was centered on the middle year (Grade 4 or 7) for computation of MLM average growth rate. 
We used a quadratic model based on previous findings (Bloom, Hill, Black, & Lipsey, 

2008) as well as inspection of the data and statistical testing of alternative growth functions. 
Because only three time points were present, the model intercept and linear slope were random 


3 


parameters but the variance of the quadratic parameter was fixed (note the omission of a residual 
term in equation 4c below) to obtain a model solution. We used two different centering 
definitions to take into account the curvilinear nature of growth. Although centering in the last, 
focal year is most consistent with the definition of other models we examined, it likely 
underestimates the amount of growth that occurs over the three year period because of 
deceleration. We therefore also centered on the middle grade in the three year span to produce 
an average growth rate over the three years. The resulting MLM model equations were: 


Level 1 (Time): 
(Yuji) = Noi + Mi (time) + m2;(time squared,j) + e:ij (3) 


Level 2 (Students): 


Toi = Pooj + Toi (4a) 

M1 = Proj + Vij (4b) 

724 = B20; (4c) 
Level 3 (Schools): 

Booj = yooo + uoo; (5a) 

B10; = y100 + U0; (5b) 

20; = y200 + U20; (5c) 


where Y;;; was the mathematics or reading/language arts scale score for student i at time ¢ in 
school j, moi was the initial status or intercept for student i at time 0 in school j, 71;; was the linear 
rate of change, m2; was the quadratic curvature representing the acceleration or deceleration in 
each student's growth trajectory and e;; was the residual for each student. At level-2, the level-1 
parameters were modeled using mean parameter values across students (/xo;) and at level-3, the 
level-2 parameters were modeled using mean parameter values across schools (xq). 


Comparison of Model Estimates 

We used several comparison criteria to evaluate the comparability and stability of school 
estimates across school performance models and across cohorts. In each state technical report 
we describe the results of our evaluation of school performance estimates. We examined: (a) 
correlations of model estimates for each school across the three cohorts, (b) correlations among 
school estimates from one model to another, (c) correlations among the school estimates and 
school composition variables (e.g., percent free/reduced lunch in the school, percent minority 
students in the school), and (d) correlations of each model with the percentage of students with 
disabilities in the school. 


Comparison of School Ranks Based on Model Estimates 

Many states and districts create school ranks based on their accountability system results. 
To compare the alternative school performance models using this metric, we created school 
percentile ranks (from | to 99, with 99 being the highest performance) based on each of the eight 
school performance model estimates described above. In one of the only studies evaluating 
school performance models, Goldschmidt, Choi, and Beaudoin (2012) compared models using 
quintiles. They examined the percentage of times schools remained in the same quintile band 
based on one school performance model versus another. Similarly, Castellano and Ho (2013) 
compared SGP and conditional regression models by examining the percentage of times schools 
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remained within 1, 5 or 10 percentile ranks for each model. To maintain some comparability 
with each of these studies, we used three levels of similarity in school ranks, computing the 
percentage of schools within 5, 10, or 20 ranks of each other. We also computed the Spearman’s 
correlation of school ranks from one cohort to another or from one school performance model to 
another. As a final comparison metric, we computed the root mean squared difference (RMSD) 
between school ranks based on each pair of cohorts or each pair of school performance models 
(see Castellano & Ho, 2013): 


Yj - (Rank je— Rank jc)? 


RMSD,- = (6) 


n 


In equation 6, for a particular school performance model, the RMSD computes the difference 
(Ranki:) between each school’s rank in one cohort (jf) versus the school’s rank in a second cohort 
(ju), squaring the difference, summing across all schools, dividing by the number of schools, n, 
and taking the square root of the result. 


RMSDmn = pica Rank jn)? (7) 


Similarly, in equation 7, the school ranks arising from alternative school performance models are 
compared in which Rankjm and Rankjn represent the rank of school j using school performance 
model m compared to that school’s rank using school performance model n. As in equation 6, 
differences in ranks are then summed, squared, divided by the number of schools and taken to 
the % power. The RMSD was a measure of similarity in school performance models where a 
lower value indicates a pair of models that rank schools most similarly. 


Summary 

We evaluated eight models for estimating school academic performance in mathematics 
and reading/language arts using operational state accountability data. In NC, OR, and PA, we 
examined stability in model estimates across three successive student cohorts in mathematics and 
reading/language arts in both elementary and middle school grades. In all four states, we also 
compared the estimates of school performance from one model to another to determine whether 
the models provided similar or different depictions of school performance, although several 
models could not be estimated in Pennsylvania because their state test did not use a vertically 
linked score scale. We then compared the degree to which model estimates correlated with 
variables that described the student composition of the school, a likely indication of construct 
irrelevant variance. Ideally estimates of school performance should not be related to the student 
composition of the school. Last, we evaluated the school performance models in terms of the 
way they ranked schools, the stability of school ranks across cohorts, and the degree of 
agreement in school rankings from one school performance model to another. Detailed results of 
these analyses and comparisons follow for the state of Oregon. 


Oregon Study 
Method 


Sample 


The initial Oregon sample consisted of all students that took the Oregon Assessment of 
Knowledge and Skills (OAKS) mathematics (NV = 483,502) or reading/language arts (N = 
474,080) general assessment in any one school year from 2007-08 through 2011-12, and whose 
records in each year were included in the state calculation of Adequate Yearly Progress (AYP). 
Students who did not follow the typical grade level sequence due to grade retention, acceleration, 
or dubious progressions were excluded from the sample; this included the transition from 
2006/07 to 2007/08, so that no students present in 2007/08 had been retained or accelerated from 
the previous year. 


The initial sample was separated into an elementary school sample (Grades 3 through 5) 
and a middle school sample (Grades 6 through 8), each consisting of three successive cohorts of 
students enrolled in school years: (a) 2007/08 through 2009/2010; (b) 2008/09 through 2010/11; 
and (c) 2009/10 through 2011/12. The initial elementary school sample for the mathematics test 
was 137,744 students. The initial middle school sample for the mathematics test was 136,535 
students. The initial elementary school sample for the reading/language arts test was 137,535 
students. The initial middle school sample for the reading/language arts test was 137,343 
students. To create an analytic sample that was appropriate for our research questions, we only 
included students with valid test scores in all three years, schools that served all three grades 
(Grades 3 through 5 or 6 through 8) for a cohort, and schools with N > 10 students in each of the 
three cohorts in the final reference year of the three-year grade level band (i.e., Grade 5 for 
elementary grades 3 to 5 and Grade 8 for middle grades 6 to 8). Students and schools that did 
not meet these criteria were excluded from analyses. As is the case in most operational and 
research applications of these models, we made no attempt to account for student mobility in 
years prior to the focal year or to make any attributions of “school effects” based on how many 
years the student had been in the focal year school. Our concern in creating the analytic sample 
was to maximize the interpretation of comparisons of the models rather than to ensure complete 
representativeness of the samples. These inclusion rules were applied to ensure that there were 
no differences in the analytic samples for different school models so that comparisons of school 
models were a function only of differences in the models and not the composition of the sample 
analyzed. The final elementary school analytic sample for the mathematics test was 90,679 
students (65.8% of the initial sample). The final middle school analytic sample for the 
mathematics test was 75,318 students (55.2%). The final elementary school analytic sample for 
the reading/language arts test was 89,627 students (65.2%). The final middle school analytic 
sample for the reading/language arts test was 75,193 students (54.7%). 


Table 1 provides summary statistics describing the school-level analytical samples of 
Oregon elementary and middle school students in the three cohorts for mathematics and 
reading/language arts. Although variation existed from cohort to cohort in sample demographic 
characteristics, generally the composition of the samples was quite similar across the three 
cohorts and for mathematics and reading/language arts at each grade level band. From 
elementary to middle school cohorts, there were small but consistent decreases in the proportion 
of English learners (EL), economically disadvantaged students (EDS), racial/ethnic minority 
students (i.e., American Indian/Alaskan Native, Asian/Pacific Islander, Black/African American, 


6 


Hispanic, Multi-Ethnic, and Declined to report), and students with disabilities (SWD). At the 
elementary school level, almost 10% of the students were EL, almost 50% of the students were 
female, over 50% were EDS, approximately 33% were racial/ethnic minority students, and about 
13% were SWD. It is also noteworthy that there was much greater school level variation—as 
indicated by the values of the standard deviations in parentheses—in EDS and racial/ethnic 
minority student school composition than other student characteristics. It should also be noted 
that when we refer to “school” composition, it references variables representing a particular 
cohort in each school in our analytic samples. Because we excluded students and schools to 
create our analytic samples, “total school” characteristics may differ slightly from the variables 
reported here. 


Table 1 


Proportion and Standard Deviation (in parentheses) of Student Subgroups for the Oregon 
Analytical Samples by Content Area and Grade Level Band 


Cohort 
1 2 3 

Mathematics Elementary EL 0.102 0.097 0.093 
(0.136) (0.129) (0.120) 

Female 0.495 0.496 0.496 

(0.093) (0.086) (0.078) 

EDS 0.514 0.523 0.543 

(0.262) (0.251) (0.247) 

Ethnic 0.323 0.332 0.343 

Minority (0.217) (0.218) (0.216) 

SWD 0.127 0.131 0.127 

(0.065) (0.068) (0.063) 

Reading/Language Arts EL 0.093 0.096 0.093 
Elementary (0.121) (0.129) (0.119) 
Female 0.497 0.497 0.497 

(0.093) (0.087) (0.078) 

EDS 0.511 0.523 0.542 

(0.262) (0.251) (0.247) 

Ethnic 0.316 0.333 0.344 

Minority (0.211) (0.218) (0.216) 

SWD 0.123 0.127 0.123 

(0.064) (0.067) (0.062) 

Mathematics Middle EL 0.056 0.048 0.034 
(0.083) (0.067) (0.052) 

Female 0.496 0.499 0.494 

(0.085) (0.091) (0.090) 

EDS 0.483 0.492 0.511 


(0.237) (0.228) (0.227) 

Ethnic 0.307 0.321 0.330 
Minority (0.229) (0.220) (0.219) 
SWD 0.116 0.113 0.116 
(0.065) (0.061) (0.068) 

Reading/Language Arts Middle EL 0.056 0.047 0.033 
(0.083) (0.066) (0.050) 

Female 0.495 0.500 0.494 
(0.087) (0.092) (0.090) 

EDS 0.482 0.491 0.510 
(0.238) (0.228) (0.227) 

Ethnic 0.307 0.319 0.330 
Minority (0.229) (0.220) (0.218) 
SWD 0.115 0.111 0.114 
(0.065) (0.060) (0.066) 


Instrument 

The outcome measures for all analyses were the standardized Oregon Assessment of 
Knowledge and Skills (OAKS; Oregon Department of Education [ODE], 2012a) mathematics 
and reading/language arts tests. The OAKS is a summative, computer-adaptive assessment 
based on the Oregon content standards (ODE, 2008). OAKS test specifications varied by grade 
and content area and were intended to measure the core content standards in the state curriculum 
(ODE, 2012a). The tests were administered under standardized conditions (ODE, 2012b). 
OAKS raw scores were converted to scale scores based on the number of items answered 
correctly while taking item difficulty into account using one parameter item response theory 
(IRT) methods and a vertical linking design over grades to create a developmental scale score 
(ODE, 2009). 

Results and Discussion 


This technical report is organized in three sections: Section A describing school 
performance model estimates, Section B describing school ranks, and Appendices with 
additional detailed results. 


Section A: School Performance Estimates 

Cohort stability. We first considered the stability of model estimates by computing the 
correlations among estimates across the three successive cohorts of students. It should be noted 
that cohort comparisons are both an indication of changes in the composition of students in the 
school from one academic year to another as well as any other temporal changes that occur from 
one year to another including changes in policy, practice, instruction, or other factors that impact 
student test scores. Table 2 shows the correlation of model estimates across cohorts for 
mathematics and reading/language arts in the elementary school and middle school samples. As 
can be seen in Table 2, correlations generally ranged only from small to moderate for the model 
estimates (with the exception of the MLMO estimates) indicating some substantial instability in 
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school performance estimates across cohorts. Correlations between adjacent years in the first 
two columns (cohort | with 2 or 2 with 3) are generally somewhat higher than the comparisons 
across two years (cohort | with 3). Although there is also some variation from elementary to 
middle school or from mathematics to reading/language arts, trends in cohort stability were fairly 


Table 2 


Correlations of School Performance Model Estimates across Cohorts by Content Area and 
Grade Level Band 


Elementary Schools 


Mathematics Reading/Language Arts 


Model Ilwith2 2with3 1 with3 lwith2 2with3 1 with3 


PP 0.593 0.630 0.529 0.655 0.667 0.593 
MLMO 0.782 0.815 0.732 0.823 0.829 OeaT 
Gain 0.369 0.305 0.174 0.281 0.261 0.084 
TM 0.319 0.267 0.139 0.208 0.093 0.115 
SGP 0.416 0.432 0.237 0.364 0.279 0.204 
VAM 0.443 0.468 0.275 0.396 0.312 0.215 
Grate 0.325 0.241 0.145 0.211 0.123 0.113 
AvGrate 0.432 0.523 0.261 0.420 0.397 0.272 


Middle Schools 
Mathematics Reading/Language Arts 


Model lwith2 2with3 1 with3 lwith2 2with3 1 with3 


PP 0.713 0.670 0.586 0.674 0.668 0.672 
MLMO _ 0.823 0.788 0.770 0.848 0.812 0.840 
Gain 0.300 0.299 0.212 0.294 0.255 0.246 
TM 0.321 01259 0.205 0.131 -0.006 0.090 
SGP 0.399 0.429 0.221 0.338 0.176 0.153 
VAM 0.414 0.491 0.269 0.308 0.314 0.171 
Grate 0.233 0.228 0.211 0.282 0.376 0.368 
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AvGrate 0.440 0.550 O237 0.542 0.465 0.248 


similar across content area and grade level band. To facilitate interpretation of the cohort results, 
we also averaged correlations across the two content areas and grade levels (see Table 3). It can 
be seen that the correlations across cohorts were largest for the two status based school 
performance measures (PP and MLMO) and noticeably lower for all other models that used two 
or three years of data to estimate school performance. The two rightmost columns of Table 3 
show the overall mean and standard deviation across the cohort comparisons for each school 
performance model. It can be seen that the greatest agreement over cohorts, content, and grade 
level was for the MLMO estimates (MLM focal year intercepts), closely followed by the PP 
model estimates. All remaining multi-year performance models had much greater instability. 
The standard deviations of correlations across cohort comparisons shown in the rightmost 
column of Table 3 also show the least variability over cohorts for the status models and the 
greatest variability across cohort correlations for the Average Growth Rate (AvGrate) model 
followed by the SGP model. 


Table 3 


Average Correlations across Content Area and Grade Level Band and Overall Mean and 
Standard Deviation (SD) Across the Three Cohort Comparisons 


Model I1with2 2with3 1 with3 Mean SD 


PP 0.659 0.659 0.595 0.638 0.040 
MLMO 0.819 0.811 0.785 0.805 0.026 
Gain 0.311 0.280 0.179 0.257 0.071 
TM 0.245 0.153 0.137 0.178 0.071 
SGP 0.379 0.329 0.204 0.304 0.100 
VAM _ 0.390 0.396 0232 0.339 0.097 
Grate 0.263 0.242 0.209 0238. ~0:052 
AvGrate 0.458 0.484 0.254 0.399 0.131 


Mean 0.440 0.419 0.324 -- = 


Comparison of models. We next computed the correlations of school performance 
estimates from one model to another within each of the three cohorts and then took the mean 
correlation across cohorts. Correlations of model estimates within each individual cohort are 
presented in Appendix A. Table 4 shows model correlations for mathematics and 
reading/language arts in the elementary school and middle school samples averaged over the 
three cohorts. 


Table 4 


Correlations of School Performance Estimates across Models by Content Area and Grade Level 
Band and Averaged over Content and Grade Band 
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Elementary School Mathematics 


Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 0.826 0.260 0.358 0.495 0.516 0.169 0.383 
MLMO 0.276 0.316 0.55 0.592 0.179 0.426 
Gain 0.894 0.849 0.872 0.960 0.573 
T™™ 0.812 0.814 0.843 0.549 
SGP 0.957 0.717 0.803 
VAM 0.736 0.843 
Grate 0.347 
Elementary School Reading/Language Arts 
Model MLMO Gain TM SGP VAM_ Grate AvGrate 
PP 0.844 -0.015 0.260 0.448 0.519 -0.013  -0.044 
MLMO -0.107 0.196 0.445 0.511 -0.120  -0.136 
Gain 0.423 0.481 0.499 0.442 0.555 
T™™ 0.709 0.716 0.672 0.406 
SGP 0.913, 0.535 0.587 
VAM 0.576 0.629 
Grate 0.274 
Middle School Mathematics 
Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 0.785 0.456 0.572 0.513 0.508 0.431 0.381 
MLMO 0.467 0.471 0.512 0.547 0.479 0.372 
Gain 0.887 0.88 0.909 0.952 0.643 
T™™ 0.811 0.809 0.836 0.577 
SGP 0.948 0.762 0.840 
VAM 0.803 0.884 
Grate 0.448 
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Middle School Reading/Language Arts 


Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 0.853 -0.139 0.360 0.395 0.399 -0.308 = -0.143 
MLMO -0.296 0.226 0.342 0.368 -0.438 = -0.250 
Gain 0.678 0.661 0.675 0.916 0.602 
T™ 0.732 0.719 0.527 0.384 
SGP 0.897 0.437 0.598 
VAM 0.499 0.682 
Grate 0.452 

Average over Content Area and Grade Level Band 

Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 0.827 0.141 0.388 0.463 0.486 0.070 0.144 
MLMO 0.085 0.302 0.462 0.505 0.025 0.103 
Gain 0.721 0.718 0.739 0.818 0.593 
TM 0.766 0.765 0.720 0.479 
SGP 0.929 0.613 0.707 
VAM 0.654 0.760 
Grate 0.380 


As can be seen in Table 4, substantial variability was present in the degree to which 
school performance estimates for one model were related to other models and the correlations 
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among models varied a good deal depending on content area and grade level band. For example, 
the correlation between the PP model and the Gain score model ranged from -.139 to +.456 and 


between MLMO and MLM Grate ranged from -.438 to +.479. The least variation in model 
correlations across content area and grade level band was for the SGP and VAM models from 


+.897 to +.957. 


As shown in the last panel of Table 4, on average across content area and grade level 
band, the highest correlations were among the SGP and VAM models (+.929), the PP and 
MLMO models (+.827), and the Gain score model with the Grate model (+.818). The lowest 
correlations were between the MLM intercept (MLMO) and MLM growth rate (Grate) models 


(+.025), the MLMO model and the Gain model (+.085), and the MLMO and the average growth 
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rate (AvGrate) model (+.103). The average correlation of the two status models (PP, MLMO) 
with the remaining six multiyear models was only +.264. Average correlations among the six 
multiple year models ranged from +.479 to +.766 with one exception, the correlation of the Grate 
and AvGrate models was only +.380, with an average correlation among all six multiyear models 
of +.691. 


We also examined the degree to which school performance model estimates were 
consistent from one content area to the other. Table 5 shows model estimate agreement across 
content areas in each cohort as well as the average across the three cohorts. As can be seen in 
Table 5, correlations were higher between content areas in elementary than middle school. On 
average, correlations for the two status models (PP and MLMO) were greater than +.70 and 
higher than average correlations for the other models that ranged from +.183 to +.578. 


Table 5 


Correlations of School Performance Model Estimates between Mathematics and 
Reading/Language Arts by Grade Level Band in each Cohort and Averaged over Cohorts 


Elementary Schools Middle Schools 
Cohort Cohort 
Model 1 2 3 Mean 1 2 3 Mean 


PP 0.772 0.753 0.761 0.762 | 0.725 0.720 0.702 0.716 
MLMO 0.865 0.852 0.845 0.854 | 0.819 0.773 0.775 0.789 
Gain 0.563 0.508 -0.122 0.316} 0.261 0.326 0.281 0.289 
TM 0.454 0.448 0.390 0.431 | 0.217 0.370 0.304 0.297 
SGP 0.558 0.503 0.513 0.525 | 0.417 0.401 0.373 0.397 
VAM _ 0.585 0.585 0.565 0.578 | 0.439 0.356 0.473 0.423 
Grate 0.582 0.468 0.385 0.478 | 0.188 0.215 0.145 0.183 
AvGrate 0.539 0.593 0.511 0.548 | 0.407 0.434 0.452 0.431 


Relation with school composition variables. We computed the correlation of model 
estimates with school composition variables to determine whether estimates were related to the 
aggregated student characteristics in each school. Table 6 shows the correlations of model 
estimates with school composition variables for mathematics and reading/language arts in the 
elementary school and middle school samples. Correlations of model estimates with school 
composition variables within each individual cohort are presented in Appendix B. 


The rightmost column of Table 6 shows the average correlation of each school 
performance model with the school composition variables across all school composition 
variables. As can be seen, correlations of the status models, PP and MLMO, were negative and 
noticeably higher than the correlations of the other school performance models with school 
composition variables. On average across content and grade level band, the correlation of the 
school composition variables was -0.221 for the PP model and -0.225 for the MLMO model. In 
contrast, the average correlations of the school composition variables with the remaining models 
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were quite low ranging from -0.003 to +0.117. Thus there was relatively little relation of the 
multiyear models with school composition, but for the status models performance estimates were 
higher the fewer the number of students from protected groups present in the school and lower as 
the number of students from protected groups increased. No clear pattern was present for the 
relation between school size and model estimates. 


Table 6 


Correlations of Model Estimates with School Composition Variables by Content Area and Grade 
Level Band 


Elementary School Mathematics 


Ethnic School 
Model EDS EL SWD- Female Minority Size Mean 


PP -0.500 -0.328 -0.194 0.013 -0.319 0.147 -0.197 
MLMO~ -0.626 -0.339 -0.195  -0.012 -0.299 0.228 = -0.207 
Gain -0.016 0.052 0.017 — -0.003 0.021 0.099 0.028 
TM -0.058 0.031 -0.016 = -0.006 -0.011 0.097 0.006 
SGP -0.158 -0.016 -0.036  -0.009 -0.028 0.171 -0.013 
VAM -0.180 -0.024 -0.052 0.006 -0.035 0.187 -0.016 
Grate -0.013 0.032 0.029  -0.006 -0.003 0.062 0.017 
AvGrate -0.020 0.090 -0.028 0.027 0.083 0.170 0.053 


Elementary School Reading/Language Arts 


Ethnic School 
Model EDS EL SWD Female Minority Size Mean 


PP -0.600 -0.526 -0.204 0.002 -0.492 0.064 -0.293 
MLMO -0.725 -0.526 -0.189 0.001 -0.477 = 0.132 -0.297 
Gain 0.158 0.184 0.076 — -0.024 0.117 0.067 0.096 
TM -0.072 0.051 0.035 0.002 0.008 0.084 0.018 
SGP -0.210 -0.061 -0.041 — -0.003 -0.105 0.110 -0.052 
VAM -0.259 -0.110 -0.059 0.009 -0.141 0.125 -0.072 
Grate 0.142 0.130 0.054 0.023 0.090 0.005 0.074 
AvGrate 0.237 0.285 0.057 = -0.002 0.210 0.052 0.140 


Middle School Mathematics 


Ethnic School 
Model EDS EL SWD- Female Minority Size Mean 


PP = -0.329 = -0.229 = -0.267 0.051 -0.265 0.169 = -0.145 
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MLMO~ -0.466 -0.232 -0.179 0.009 -0.184 0.236 = -0.136 
Gain -0.009 0.017  -0.070 0.030 0.056 0.140 0.027 
TM -0.046 -0.001 -0.118 0.037 0.005 0.166 0.007 
SGP 0.005 0.045 -0.049 0.026 0.060 0.172 0.043 
VAM 0.006 0.045 -0.048 0.020 0.063 0.181 0.044 
Grate -0.090 -0.045  -0.074 0.011 -0.001 0.138  -0.010 
AvGrate 0.134 0.128 0.001 0.020 0.113 0.150 0.091 


Middle School Reading/Language Arts 


Ethnic School 
Model EDS EL SWD Female Minority Size Mean 


PP -0.525 -0.444 -0.263 0.135 -0.481 0.088  -0.248 
MLMO -0.663 -0.459 -0.217 0.120 -0.456 0.109 -0.261 
Gain 0.326 0.238 0.092  -0.026 0.226 0.031 0.148 
TM -0.021 0.001 -0.001 0.036 -0.028 0.112 0.017 
SGP -0.063 0.016 -0.035 0.056 -0.047 0.128 0.009 
VAM -0.058 0.002 -0.038 0.041 -0.058 0.121 0.002 
Grate 0.365 0.256 0.126  -0.054 0235 0.007 0.156 
AvGrate 0.394 0.339 0.098  -0.039 0.258 0.057 0.185 


Relation of Model Estimates to SWD School Composition. Because of the NCAASE 
emphasis on the performance and academic growth of SWD, we also focused more specifically 
on the relations between the percentage of SWD students served by a school and the school 
performance model estimates. Correlations of model estimates with SWD school composition 
within each individual cohort are presented in Appendix C. Table 7 shows the correlation of 
model estimates with the percentage of SWD in each school for mathematics and 
reading/language arts in the elementary school and middle school samples averaged over 
cohorts. As can be seen in the bottom row of Table 7, average school performance estimates 
based on the single-year, status models (PP and MLMO) had substantially higher correlations 
with school SWD composition than the other school performance models. With the PP and 
MLMO models, school performance estimates were higher the lower the percentage of SWD 
students in the school and lower to the extent that the school served higher proportions of SWD. 


Table 7 


Average School Performance Model Estimates as a Function of the Percentage of SWD in the 
School by Content Area and Grade Level Band 


Content Area and 
Grade Level Band PP MLMO ~~ Gain ™ SGP VAM Grate AvGrate 


Math Elementary -0.194 -0.195 0.017  -0.016 -0.036 -0.052 0.029 = -0.028 
Reading/Language -0.204 -0.189 0.076 0.035 -0.041 -0.059 0.054 0.057 
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Arts Elementary 
Math Middle -0.267 =-0.179 -0.070 -0.118 -0.049 -0.048 -0.074 0.001 


Reading/Language -0.263 -0.217 0.092 -0.001 -0.035 -0.038 0.126 0.098 
Arts Middle 


Mean -0.232 -0.195 0.029 -0.025 -0.040 -0.049 0.034 0.032 


Summary of Section A. We evaluated eight alternative models for estimating school 
academic performance in mathematics and reading/language arts using operational Oregon state 
accountability data. We observed substantial variability in model estimates across three 
successive student cohorts in mathematics and reading/language arts in both elementary and 
middle school grades. Variability across cohorts was somewhat less for the two status models 
(PP and MLM0O) than for the remaining models that used more than one year of data. We also 
compared the estimates of school performance from one model to another and found substantial 
disagreement across models. Generally, the status models based on a single year of data were 
more similar to each other and differed from the remaining models that examined more than one 
year of data. There was greater agreement among the models that used multiple years of data. 
We also compared school performance estimates in mathematics with those in reading/language 
arts. Again, agreement was greater across content areas for the status models than for the 
multiple year models. Comparison of model estimates with school composition variables 
showed that status models (PP and MLMO) had substantially higher correlations than the 
remaining school performance models with the student makeup of the school with lower 
estimates related to higher proportions of protected student subgroups in the school. Finally, we 
correlated school performance estimates with the percentage of SWD in each school. Ideally 
estimates of school performance should be unrelated to the student composition of the school, 
but as with the other school composition variables, we found that the status models were more 
highly correlated with SWD school composition and noticeably lower correlations of the 
multiyear model estimates with the percentage of SWD students in the school. 


Section B: School Ranks Based on School Performance Estimates 

The purpose of Section B of this technical report is to describe study results based on 
school ranks. In section A, we focused on the examination of school performance estimates that 
arose from the eight school performance models. In section B, we focus on the examination of 
school ranks based on those school performance estimates. The technical report overview 
(http://www.ncaase.com/publications/tech-reports) describes the methods for analyzing school 
ranks, comparing school ranks across cohorts, and comparing school performance models using 
school ranks. Common practice for states and other jurisdictions is to rank schools as a method 
for evaluating academic performance. Therefore, using the estimates of school performance 
generated by the eight models described previously, we computed percentile ranks for each 
school. We then compared school ranks within each school performance model across the three 
cohorts used in the study. Next we compared the school ranks for each model to the ranks 
obtained from each of the other models. Finally we examined the relation between school ranks 
from each model with variables describing the student composition of each school. Three 
criteria were used to evaluate the comparisons of school ranks: (a) the Spearman’s correlation 
between school ranks, (b) the proximity of absolute school ranks, and (b) the root mean square 
difference (RMSD) in school ranks (see technical report Overview for further details). 
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Comparison of cohorts. We first consider the stability of school ranks within each 
school performance model across the three successive cohorts of students in mathematics and 
reading/language arts in the elementary and middle school grades. We computed the Spearman’s 
correlation of the school ranks from one cohort to the school ranks from each of the other two 
cohorts within each of the eight school performance models to determine the stability of school 
ranks. As mentioned in Section A, cohort comparisons are both an indication of changes in the 
composition of students in the school from one academic year to another as well as any other 
temporal changes that occur from one year to another including changes in policy, practice, 
instruction, or other factors that impact student test scores. Table 8 shows the correlation of 
school ranks across cohorts for mathematics and reading/language arts in the elementary school 
and middle school samples. As can be seen in Table 8, the correlations generally ranged from 
small to moderate indicating substantial variability in school ranks from one cohort to another. 
As would be expected, correlations between adjacent years in the first two columns (cohort 1 
with 2 or 2 with 3) were generally somewhat higher than the comparison across two years 
(cohort 1 with 3). Although there was some variation, results were quite similar from elementary 
to middle school or from mathematics to reading/language arts. 


Table 8 


Spearman's Correlations of Model School Ranks for Each Pair of Cohorts by Content Area and 
Grade Level Band 


Elementary Schools 


Mathematics Reading/Language Arts 


Model Ilwith2 2with3 1 with3 lwith2 2with3 1 with3 


PP 0.620 0.613 0.529 0.658 0.659 0.609 
MLMO _ 0.748 0.787 0.693 0.812 0.811 0.787 
Gain 0.349 0.262 0.141 0.240 0.247 0.070 
TM 0.315 0.255 0.117 0.173 0.116 0.102 
SGP 0.406 0.409 0.217 0.354 0.282 0.193 
VAM 0.428 0.450 0.235 0.388 0.324 0.215 
Grate 0.297 0.195 0.124 0.195 0.120 0.120 
AvGrate 0.419 0.512 0.242 0.424 0.384 0.248 
Middle Schools 
Mathematics Reading/Language Arts 


Model 1with2 2with3 1 with3 lwith2 2with3 1 with3 


PP = 0.675 0.635 0.511 0.707 0.686 0.684 
17 
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MLMO _ 0.754 0.730 0.700 0.837 0.797 0.824 
Gain 0.269 0.267 0.167 0.263 0.304 0.281 
T™ 0.306 0;222 0.171 0.096 0.121 0.086 
SGP 0.373 0.418 0.197 0.323 0.243 0.162 
VAM — 0.402 0.452 0.238 0.295 0.301 0.158 
Grate 0.230 0.207 0.169 0.272 0.370 0.373 
AvGrate 0.442 0.512 0.201 0.516 0.456 0.261 


To facilitate further interpretation, we averaged the results shown in Table 8 across 
content area and grade level band. As can be seen in Table 9, on average the greatest stability 
was for the two status models, PP and MLMO. Noticeably lower correlations occurred for the 
remaining school performance models, all of which were based on more than one year of data, 
with the least stability for the TM, Grate, and Gain models. 


Table 9 


Spearman's Correlations of Model School Ranks Averaged across Content Area and Grade 
Level Band and Overall Mean and Standard Deviation (SD) Across the Three Cohort 
Comparisons 


Model I1with2 2with3 1 with3 Mean SD 


PP 0.665 0.648 0.583 0.632 0.044 
MLMO 0.788 0.781 0.751 0.773 0.027 
Gain 0.280 0.270 0.165 0.238 0.071 
TM 0.222 0.178 0.119 0.173 0.056 
SGP 0.364 0.338 0.192 0.298 0.097 
VAM 0.378 0.382 0.212 0.324 0.100 
Grate 0.248 0.223 0.196 0.222 0.055 
AvGrate 0.450 0.466 0.238 0.385 0.131 


Our second criterion for comparing school ranks was to determine how much a school’s 
rank changed from one cohort to another. Table 10 shows the proportion of schools that were 
within 5, 10, or 20 ranks in one cohort versus another for each school performance model in 
mathematics and reading/language arts at each grade level band. The last table entry for each 
school performance model shows the average differences in school ranks averaged over content 
area and grade level band. It can be seen that on average for the PP model, about one quarter of 
the schools differed by only 5 percentile ranks or less, slightly over 40% of schools differed by 
10 ranks or less, and slightly more than 60% differed by 20 ranks or less. This also indicates that 
over a third of schools differed by more than 20 ranks from one cohort to another. The results 
for the MLMO model showed slightly greater agreement in school ranks across cohorts. 
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However, the level of agreement in school ranks across cohorts was noticeably lower for all of 
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the remaining models that were based on two or more years of achievement data. For example, 


school ranks based on the remaining models (Gain, TM, SGP, VAM, Grate, and AvGrate) 


differed by more than 20 ranks for about 50% or more of the schools. 


Table 10 


Proportion of Elementary or Middle Schools Within 5, 10, or 20 Ranks of Each Other for each 
School Performance Model for each Pair of Cohorts in Mathematics and Reading/Language 


Arts 
PP 
Cohort r=5 r=10 r=20 
Mathematics Elementary lvs.2 0.229 0.380 0.642 
2vs.3 0.236 0.383 0.617 
lvs.3 0.228 0.376 0.589 
Reading/Language Arts Elementary Il vs.2 0.267 0.437 0.652 
2vs.3 0.225 0.389 0.652 
lvs.3 0.219 0.387 0.629 
Mathematics Middle lvs.2 0.255 0.420 0.646 
2vs.3 0.272 0.399 0.638 
lvs.3 0.259 0.395 0.580 
Reading/Language Arts Middle lvs.2 0.284 0.453 0.741 
2vs.3 0.309 0.490 0.700 
lvs.3 0.259 0.465 0.695 
Mean lvs.2 0.259 0.422 0.670 
2vs.3 0.260 0.415 0.652 
lvs.3 0.241 0.406 0.623 
MLMO 
Cohort r=5 r=10 r=20 
Mathematics Elementary lvs.2 0.289 0.452 0.718 
2vs.3 0.309 0.512 0.752 
lvs.3 0.262 0.421 0.685 
Reading Elementary lvs.2 0.288 0.520 0.773 
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2vs.3 0.325 0.525 0.783 
lvs.3 0.296 0.508 0.724 
Mathematics Middle lvs.2 0.292 0481 0.691 
2vs.3 0.267 0.436 0.695 
lvs.3 0.272 0.440 0.671 
Reading Middle lvs.2 0.354 0.593 0.811 
2vs.3 0.374 0.539 0.790 
lvs.3 0.342 0.523 0.798 
Mean lvs.2 0.306 0.511 0.748 
2vs.3 0.319 0.503 0.755 
lvs.3 0.293 0.473 0.720 
Gain 
Cohort r=5 r=10 r=20 
Mathematics Elementary 1lvs.2 0.158 0.295 0.487 
2vs.3 0.144 0.261 0.467 
lvs.3 0.140 0.243 0.422 
Reading Elementary lvs.2 0.142 0.250 0.440 
2vs.3 0.171 0.283 0.447 
Ilvs.3 0.141 0.233 0.406 
Mathematics Middle lvs.2 0.165 0.284 0.453 
2vs.3 0.148 0.267 0.486 
Ilvs.3 0.119 0.235 0.412 
Reading Middle lvs.2 0.173 0.309 0.486 
2vs.3 0.173 0.292 0.498 
lvs.3 0.136 0.305 0.490 
Mean lvs.2 0.160 0.284 0.466 
2vs.3 0.159 0.276 0.475 
lvs.3 0.134 0.254 0.432 
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Cohort r=5 r=10 r=20 
Mathematics Elementary lvs.2 0.155 0.276 0.500 
2vs.3 0.144 0.248 0.459 
lvs.3 0.127 0.249 0.452 
Reading Elementary lvs.2 0.131 0.233 0.445 
2vs.3 0.123 0.235 0.440 
Ilvs.3 0.118 0.212 0.394 
Mathematics Middle lvs.2 0.152 0.280 0.494 
2vs.3 0.165 0.276 0.490 
lvs.3 0.165 0.247 0.432 
Reading Middle lvs.2 0.128 0.243 0.432 
2vs.3 0.132 0.247 0.477 
lvs.3 0.128 0.235 0.383 
Mean lvs.2 0.142 0.258 0.468 
2vs.3 0.141 0.252 0.466 
lvs.3 0.134 0.236 0.415 
SGP 
Cohort r=5 r=10 r=20 
Mathematics Elementary Il vs.2 0.168 0.285 0.495 
2vs.3 0.191 0.299 0.500 
lvs.3 0.150 0.256 0.446 
Reading Elementary lvs.2 0.179 0.278 0.490 
2vs.3 0.149 0.273 0.480 
lvs.3 0.151 0.243 0.424 
Mathematics Middle lvs.2 0.206 0.321 0.502 
2vs.3 0.156 0.313 0.568 
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lvs.3 0.177 0.259 0.420 
Reading Middle lvs.2 0.169 0.305 0.481 
2vs.3 0.132 0.272 0.490 
Ilvs.3 0.119 0.276 0.395 
Mean lvs.2 0.180 0.297 0.492 
2vs.3 0.157 0.289 0.509 
Ilvs.3 0.149 0.258 0.421 
VAM 
Cohort r=5 r=10 r=20 
Mathematics Elementary 1vs.2 0.172 0.305 0.508 
2vs.3 0.168 0.305 0.520 
lvs.3 0.147 0.243 0.444 
Reading Elementary lvs.2 0.159 0.262 0.498 
2vs.3 0.159 0.280 0.480 
Ilvs.3 0.131 0.230 0.419 
Mathematics Middle lvs.2 0.198 0.309 0.510 
2vs.3 0.218 0.325 0.510 
lvs.3 0.140 0.243 0.449 
Reading Middle lvs.2 0.169 0.276 0.469 
2vs.3 0.152 0.263 0.473 
lvs.3 0.169 0.251 0.395 
Mean lvs.2 0.174 0.288 0.496 
2vs.3 0.174 0.293 0.496 
lvs.3 0.147 0.242 0.427 
Grate 
Cohort r=5 r=10 r=20 
Mathematics Elementary lvs.2 0.168 0.272 0.467 
2vs.3 0.145 0.257 0.452 
lvs.3 0.117 0.231 0.394 
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Reading Elementary lvs.2 0.139 0.255 0.442 
2vs.3 0.121 0.224 0.389 
lvs.3 0.137 0.240 0.417 
Mathematics Middle lvs.2 0.144 0.247 0.465 
2vs.3 0.169 0.280 0.420 
Ivs.3 0.132 0.259 0.416 
Reading Middle lvs.2 0.181 0.296 0.502 
2vs.3 0.165 0.305 0.531 
lvs.3 0.152 0.313 0.473 
Mean lvs.2 0.158 0.268 0.469 
2vs.3 0.150 0.266 0.448 
Ivs.3 0.134 0.261 0.425 
AvGrate 
Cohort r=5 r=10 r=20 
Mathematics Elementary lvs.2 0.147 0.271 0.507 
2vs.3 0.173 0.325 0.561 
lvs.3 0.145 0.254 0.457 
Reading Elementary lvs.2 0.172 0.295 0.517 
2vs.3 0.137 0.265 0.470 
lvs.3 0.154 0.255 0.444 
Mathematics Middle lvs.2 0.169 0.313 0.490 
2vs.3 0.206 0.317 0.543 
Ivs.3 0.115 0.222 0.370 
Reading Middle lvs.2 0.198 0.342 0.568 
2vs.3 0.165 0.296 0.539 
lvs.3 0.136 0.280 0.490 
Mean lvs.2 0.172 0.305 0.520 
2vs.3 0.170 0.301 0.528 
lvs.3 0.138 0.253 0.440 
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Our third criterion for comparing school ranks was to calculate the root mean square 
difference (RMSD) between cohorts or models as defined in the report overview. Table 11 
shows the RMSD across pairs of cohorts by content area and grade level band for each of the 
eight school performance models and in the last two columns the mean and standard deviation 
(SD) across cohort comparisons. As can be seen in the table, the smallest differences in rank 
were for the MLMO model, about 17 to 21 ranks on average, followed by the PP model. 
Average differences in school rank across cohorts for the remaining models ranged from about 
31 to 38. 


Table 11 


RMSD in School Ranks for each Student Cohort for each School Performance Model by Content 
Area and Grade Level Band 


Elementary School Mathematics 


Model lwith2 2with3 I1with3 Mean SD 


PP 24.884 25.134 27.684 25.901 1.549 
MLMO 20.269 18.610 22.359 20ATIW  T87t 
Gain 32.505 34.659 37.401 34.872 2.430 
TM 33.382 34.818 31,929. 35.376. 2.324 
SGP 31.076 31.011 35.696 32.594 2.686 
VAM 30.527 29.940 35.304 31.924 2.942 
Grate 33.823 36.204 37.772 35.933 1.988 
AvGrate 30.753 28.179 39139 31.357 3.319 


Elementary School Reading/Language Arts 


Model lwithh2 2with3 Iwith3 Mean SD 


PP 23.598 23,092 25.257 24.149 0.960 
MLMO 17.507 17.542 18.631 17.893 0.639 
Gain 35.183 35.019 38.914 36.372 2.203 
TM 36.703 37.928 38.240 37.624 0.812 
SGP 32.430 34.189 36.250 34.290 1.912 
VAM 31.568 33.183 35.764 33.505 2.116 
Grate 36.207 37.847 37.852 37.302 0.948 
AvGrate 30.618 31.678 35.001 32.432 2.287 


Middle School Mathematics 


Model Iwith2 2with3 1lwith3 Mean SD 


PP 22.959 = 24.334 =. 28.150) 25.148 2.689 
MLMO- 19.945 20.889 = 22.024 20.953 1.041 
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2) 


Gain 34.406 34.469 36.731 35.202 1.325 
TM 33.549 = 335.508 = 36.662 35.240 —-1.574 
SGP 31.858 = 30.695. 36.080 32.878 2.834 
VAM 31.125, 29.782 35.122 32.010 2.778 
Grate 35.309 35.846 36.690 35.948 0.696 
AvGrate 30.048 = =28.129 = 35.970 31.382 = 4.087 


Middle School Reading/Language Arts 


Model lwith2 2with3 t1with3 Mean SD 


PP 21.787) = 22.540 =. 22.604 22.310 0.454 
MLMO 16.271 18.141 16.903 17.105 0.951 
Gain 34.532 33.582 34.127 34.080 0.477 
TM 38.265 =. 337.749 338.471 = 38.162 = 0.372 
SGP 33.096 =. 35.016 = 336.844 «34.985 1.874 
VAM = 33.787 —s- 333.661 = 36.927 34.792 1.851 
Grate 34.337 31.956 31.883 32.725 1.396 
AvGrate 28.016 29.695 34.590 30.767 3.416 


Comparison of models. We next compared school ranks from one model to another 
within each of the three cohorts. Comparisons of school ranks within each individual cohort were 
computed and are presented in Appendix D. We averaged those results by taking the median 
absolute difference in school ranks over the three cohorts in mathematics and reading/language 
arts in the elementary and middle school grades. For each pair of school performance models, 
Table 12 shows the average percentage of schools that were within 5, 10, or 20 percentile ranks 
in one model versus the other. As can be seen in the table, the PP and MLMO models ranked 
schools most similarly, over 50% of schools were within 10 ranks and over 80% were within 20 
ranks for these two models. The level of agreement in school ranks was much lower when 
comparing either the PP or the MLMO models with any of the other models that examined two or 
more years of achievement. Generally, either the PP or MLMO model school rankings agreed 
with the multiyear models within 20 ranks in from about 40% to 60% of schools. 


Agreement of the multiyear models with each other was greater with from about 50% to 
90% of models ranking schools within 20 ranks of each other. Some of the lowest agreement in 
ranks occurred between the average MLM growth rate (AvGrate) rankings and other models, 
ranging from about 50% to about 70% of school within 20 ranks of each other. 


Table 12 


Proportion of Elementary or Middle Schools within 5, 10, or 20 Ranks of Each Other for each 
Pair of School Performance Models in Mathematics and Reading/Language Arts Averaged over 
Cohorts 


Model Comparison: r=5 r=10 r=20 


PP vs. MLMO 
Math Elementary 0.363 0.579 0.835 
Reading/Language Arts Elementary 0.370 0.584 0.840 
Math Middle 0.354 0.56 0.822 
Reading/Language Arts Middle 0.406 0.645 0.875 
Mean 0.373 0.592 0.843 
PP vs. Gain 
Math Elementary 0.147 0.251 0.441 
Reading/Language Arts Elementary 0.110 0.204 0.356 
Math Middle 0.178 0.306 0.501 
Reading/Language Arts Middle 0.097 0.180 0.326 
Mean 0.133 0.235 0.406 
PP vs. TM 
Math Elementary 0.160 0.276 0.471 
Reading/Language Arts Elementary 0.134 0.244 0.455 
Math Middle 0.217 0.357 0.583 
Reading/Language Arts Middle 0.178 0.298 0.498 
Mean 0.172 0.294 0.502 
PP vs. SGP 
Math Elementary 0.184 0.305 0.545 
Reading/Language Arts Elementary 0.169 0.309 0.524 
Math Middle 0.202 0.339 0.540 
Reading/Language Arts Middle 0.167 0.283 0.499 
Mean 0.180 0.309 0.527 
PP vs. VAM 
Math Elementary 0.183 0.32 0.554 
Reading/Language Arts Elementary 0193". -:O:332 0.558 
Math Middle 0.193 0.325 0.534 
Reading/Language Arts Middle 0.150 0.270 0.495 
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Mean 0.180 0.312 0.535 
PP vs. Grate 
Math Elementary 0.131 0.237 0.415 
Reading/Language Arts Elementary 0.104 0.201 0.361 
Math Middle 0.177. 0.309 0.505 
Reading/Language Arts Middle 0.081 0.143 0.289 
Mean 0.123... 0.222 0.392 
PP vs. AvGrate 
Math Elementary 0.158 0.281 0.502 
Reading/Language Arts Elementary 0.100 0.188 0.357 
Math Middle 0.180 0.307 0.487 
Reading/Language Arts Middle 0.095 0.195 0.325 
Mean 0.133 0.243 0.418 
MLM0 vs. Gain 
Math Elementary 0.145 0.262 0.463 
Reading/Language Arts Elementary 0.098 0.185 0.348 
Math Middle 0.171 0.320 0.527 
Reading/Language Arts Middle 0.091 0.170 0.322 
Mean 0.126 0.234 0.415 
MLM0 vs. TM 
Math Elementary 0.144 0.273 0.480 
Reading/Language Arts Elementary 0.132 0.237 0.421 
Math Middle 0.167 0.320 0.547 
Reading/Language Arts Middle 0.151 0.267 0.447 
Mean 0.148 0.274 0.474 
MLM0 vs. SGP 
Math Elementary 0.189 0.345 0.576 
Reading/Language Arts Elementary 0.172 0.294 0.509 
Math Middle 0.189 0.339 0.554 


Zi 
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Reading/Language Arts Middle 0.166 0.270 0.508 
Mean 0.179 0.312 0.537 
MLM0 vs. VAM 
Math Elementary 0.207 =—0.353 0.595 
Reading/Language Arts Elementary 0.175 0.316 0.549 
Math Middle 0.196 0.337 0.565 
Reading/Language Arts Middle 0.141 0.278 0.514 
Mean 0.180 0.321 0.556 
MLM0 vs. Grate 
Math Elementary 0.124 0.241 0.420 
Reading/Language Arts Elementary 0.088 0.181 0.337 
Math Middle 0.203 0.342 0.551 
Reading/Language Arts Middle 0.082 0.152 0.288 
Mean 0.124 0.229 0.399 
MLM0 vs. AvGrate 
Math Elementary 0.181 0.306 0.523 
Reading/Language Arts Elementary 0.099 0.183 0.349 
Math Middle 0.181 0.309 0.483 
Reading/Language Arts Middle 0.104 0.182 0.332 
Mean 0.141 0.245 0.422 
Gain vs. TM 
Math Elementary 0.404 0.631 0.875 
Reading/Language Arts Elementary 0.215 0.361 0.588 
Math Middle 0.421 0.636 0.866 
Reading/Language Arts Middle 0.232 0.416 0.624 
Mean 0.318 0.511 0.738 
Gain vs. SGP 
Math Elementary 0.357 = 0.552 0.812 
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Reading/Language Arts Elementary 0.206 0.341 0.552 
Math Middle 0.394 0.616 0.846 
Reading/Language Arts Middle 0.251 0.383 0.630 
Mean 0.302 0.473 0.710 
Gain vs. VAM 
Math Elementary 0.356 0.585 0.836 
Reading/Language Arts Elementary 0.192 0.334 0.566 
Math Middle 0.439 0.649 0.883 
Reading/Language Arts Middle 0.222 0.368 0.649 
Mean 0.302 0.484 0.734 
Gain vs. Grate 
Math Elementary 0.583 0.823 0.98 
Reading/Language Arts Elementary 0.334 0.524 0.715 
Math Middle 0.553 0.816 0.971 
Reading/Language Arts Middle 0.468 0.739 0.938 
Mean 0.485 0.726 0.901 
Gain vs. AvGrate 
Math Elementary 0.211 0.371 0.597 
Reading/Language Arts Elementary 0.198 0.358 0.571 
Math Middle 0.265 0.399 0.626 
Reading/Language Arts Middle 0.237 = 0.373 0.617 
Mean 0.228 0.375 0.603 
TM vs. SGP 
Math Elementary 0.329 0.512 0.782 
Reading/Language Arts Elementary 0.256 0.423 0.652 
Math Middle 0.335 0.534 0.776 
Reading/Language Arts Middle 0.314 0.464 0.716 
Mean 0.308 0.483 0.732 


TM vs. VAM 
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Math Elementary 0.327 0.520 0.783 
Reading/Language Arts Elementary 0.246 0.401 0.662 
Math Middle 0.333 0.538 0.791 
Reading/Language Arts Middle 0.270 0.443 0.727 
Mean 0.294 0.476 0.741 
TM vs. Grate 
Math Elementary 0.337 0.544 0.807 
Reading/Language Arts Elementary 0.228 0.397 0.656 
Math Middle 0.368 0.584 0.822 
Reading/Language Arts Middle 0.203. 0.347 0.553 
Mean 0.284 0.468 0.710 
TM vs. AvGrate 
Math Elementary 0.201 0.356 0.575 
Reading/Language Arts Elementary 0.166 0.292 0.487 
Math Middle 0.225 0.383 0.597 
Reading/Language Arts Middle 0.163 0.298 0.498 
Mean 0.189 0.332 0/339 
SGP vs. VAM 
Math Elementary 0.575 0.822 0.974 
Reading/Language Arts Elementary 0.430 0.679 0.907 
Math Middle 0.598 0.833 0.974 
Reading/Language Arts Middle 0.425 0.700 0.920 
Mean 0.507 0.758 0.944 
SGP vs. Grate 
Math Elementary 0.256 0.439 0.678 
Reading/Language Arts Elementary 0.188 0.329 0.552 
Math Middle 0.316 0.464 0.723 
Reading/Language Arts Middle 0.177 0.298 0.506 
Mean 0.234 0.382 0.615 
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SGP vs. AvGrate 


Math Elementary 0.317 0.526 0.770 
Reading/Language Arts Elementary 0.216 0.360 0.582 
Math Middle 0.337 0.556 0.807 
Reading/Language Arts Middle 0.222 0.369 0.615 
Mean 0.273 0.453 0.694 
VAM vs. Grate 

Math Elementary 0.267 0.440 0.689 
Reading/Language Arts Elementary 0.210 0.355 0.573 
Math Middle 0.311 0.497 0.745 
Reading/Language Arts Middle 0.181 0.331 0.539 
Mean 0.242 0.406 0.636 

Grate vs. AvGrate 
Math Elementary 0.169 0.288 0.501 
Reading/Language Arts Elementary 0.138 0.256 0.438 
Math Middle 0.204 0.329 0.517 
Reading/Language Arts Middle 0.191 0.337 0.551 
Mean 0.176 0.302 0.502 


Our last criterion for comparing school ranks across cohorts was the RMSD between 
pairs of school performance model rankings. Appendix E shows the RMSD between pairs of 
school performance model rankings for each individual cohort. Table 13 shows the RMSD 
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averaged over the three cohorts by content area and grade level band. The RMSD values reflect 


the same patterns of results for models as described previously. The greatest agreement in 
average ranks was between the PP and MLMO models for which schools differed by about 16 


ranks or less on average. Much larger differences (about 30 ranks or more on average) occurred 


between the PP and MLMO versus the other school performance models. Agreement in school 
ranks between the remaining models was generally greater, in the range of 10 to 20 ranks on 
average, although differences were somewhat larger between the AvGrate ranks and school 


ranks based on other models. 


Table 13 


Average across Cohorts of RMSD in School Ranks between School Performance Models by 


Content Area and Grade Level Band 
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Elementary School Mathematics 


Model MLMO ~~ Gain TM SGP VAM_~ Grate AvGrate 
PP 14.868 34.971 33.024 29.202 28.273 36.925 32.098 
MLMO 34.243 33.067 27.245 26.140 36.416 30.701 
Gain 13.260 15.971 14.623 7.943 26.889 
T™™ 17.755 17.273) 15.952 = 27.520 
SGP 8.522: 21,691. ~18.525 
VAM 21.036 16.848 
Grate S23 03 
Elementary School Reading/Language Arts 
Model MLMO Gain T™ SGP VAM_ Grate AvGrate 
PP 14.712 41.170 34.889 30.196 28.243 40.873 41.440 
MLMO 42.355 36.270 30.064 28.287 42.658 42.537 
Gain 29.216 28.802 27.719 23.478 27.504 
T™ 22.165 22.238 23.411 31.715 
SGP 11.868 28.201 26.374 
VAM 27.033 25.169 
Grate 34.942 
Middle School Mathematics 
Model MLMO~ Gain T™ SGP VAM_ Grate AvGrate 
PP 16.219 30.145 26.924 28.58 28.258 30.096 32.242 
MLMO 28.676 27.745 27.155 26.782 28.209 31.953 
Gain 14.457 14.683 12.932 8.326 24.913 
T™™ 18.225 17.644 15.777 26.607 
SGP 8.330 20.216 16.257 
VAM 19.246 14.308 
Grate 30.695 
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Middle School Reading/Language Arts 


Model MLMO Gain ™ SGP VAM~— Grate AvGrate 


PP 13.470 43.525 33.005 31.943 31.717 46.231 43.532 


MLMO 45.499 35.079 32.500 32.284 47.907 44.730 
Gain 23.929 24.510 23.046 10.437 25.334 
T™™ 21.508 20.289 27.575 31.313 
SGP 11.374 30.572 25.692 
VAM 29.257 23.367 
Grate 30.083 


We also evaluated the extent to which school ranks agreed from one content area to the 
other. Table 14 shows the Spearman’s correlation of school ranks in mathematics with school 
ranks in reading/language arts by cohort and grade level band. The table also shows the mean 
correlation across cohorts at the two grade level bands. As can be seen in Table 14, on average 
correlations of school ranks across mathematics and reading/language arts in elementary schools 
ranged from +.301 to +.839 for the different school performance models. For middle schools, 
the average correlations ranged from +.160 to +.759. Correlations were higher for the two status 
models, and lower for the multiyear models at both grade level bands. Generally, correlations 
across content were lower for the other models. Average correlations at the middle school level 
were also consistently lower than for elementary schools for all models. 


Table 14 


Spearman's Correlations of School Performance Model Estimates across Mathematics and 
Reading/Language Arts by Cohort 


Elementary Schools Middle Schools 


Model Cohort1 Cohort2 Cohort3 Mean | Cohort1 Cohort2 Cohort3 Mean 


PP 0.781 0.755 0.755 0.764 0.716 0.693 0.659 0.689 
MLMO 0.855 0.840 0.823 0.839 0.789 0.735 0.754 0.759 
Gain 0.550 0.465 -0.112 0.301 0.243 0.282 0.232. 0252 
T™ 0.443 0.396 0.383 0.407 0.245 0.310 0.344 0.300 
SGP 0.550 0.476 0.521 0.516 0.401 0.347 0.377 0.375 
VAM 0.576 0.546 0.569 0.564 0.440 0.364 0.457 0.420 
Grate 0.572 0.441 0.356 0.456 0.173 0.186 0.122 0.160 
AvGrate 0.529 0.579 0.515 0.541 0.417 0.450 0.427 0.431 
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Table 15 shows the proportion of schools that shared similar ranks in mathematics as in 
reading/language arts for each school performance model by school level and averaged over 
grade level band. Similar to results previously described, Table 15 shows greater agreement for 
the PP and MLMO models than the other school performance models with about 70% or more of 
the schools having ranks within 20 places across grade level bands. In contrast, there was 
substantially less agreement across the two content areas for the remaining, multiyear models 
with only approximately 50% of schools agreeing within 20 ranks for most models in either 
grade level band. 


Table 15 


Proportion of Elementary or Middle Schools within 5, 10, or 20 Ranks of Each Other in 
Mathematics versus Reading/Language Arts for each School Performance Model Averaged over 
Cohorts 


Model Comparison r=5 r=10 r=20 


PP 


Elementary 0.316 0.500 0.741 
Middle 0.270 0.440 0.675 
Mean 0.293 0.470 0.708 
MLMO 


Elementary 0.339 0.547 0.805 
Middle 0.287 0.473 0.741 
Mean 0.313 0.510 0.773 
Gain 


Elementary 0.144 0.276 0.493 
Middle 0.150 0.278 0.481 
Mean 0.147 0.277 0.487 


Elementary 0.161 0.295 0.521 
Middle 0.143 0.284 0.499 
Mean 0.152 0.290 0.510 

SGP 


Elementary 0.198 0.340 0.556 
Middle 0.166 0.274 0.490 
Mean 0.182 0.307 0.523 

VAM 


Elementary 0.197 0.351 0.573 
Middle 0.162 0.292 0.495 
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3D 


Mean 0.180 0.322 0.534 
Grate 


Elementary 0.172 0.316 0.532 
Middle 0.132 0.239 0.460 
Mean 0.152 0.277 0.496 

AvGrate 


Elementary 0.190 0.339 0.564 
Middle 0.176 0.291 0.538 
Mean 0.183 0.315 0.551 


Calculation of the RMSD in school ranks for mathematics versus reading/language arts 
by cohort and grade level band and averaged over cohorts showed similar results (see Table 16). 
The difference in school ranks averaged over cohorts for the PP and MLMO models ranged from 
about 16 to 22. Average differences in rank across the two content areas were substantially 
greater for the remaining models ranging from 20 to 37 depending on model and grade level 
band. 


Table 16 


RMSD in School Ranks for Mathematics and Reading/Language Arts by Cohort and Grade Level 
Band and Overall Means and Standard Deviations (SD) 


Elementary Schools Middle Schools 


Model Cohort 1 Cohort2 Cohort 3 Mean | Cohort 1 Cohort 2 Cohort 3 Mean 


PP 18.842 19,929 19.925. 19565 21.421 2224) 23.507 22.402 
MLMO 15.348 16.091 16.953 16.131 18.494 20.713 19.947 19.718 
Gain 27.018 29.447 42.488 32.984 35.015 34.106 35.280 34.800 
T™™ 30.044 31.282 31.667 30.998 34.945 33.477 32.626 33.683 
SGP —_.27.007 29.155 27.882 28.015 31.148 32,517 31.749 31.805 
VAM 26.227 ZEST 26.445 26.603 30.117 32.090 29.670 30.626 
Grate 26.340 30.094 32:333. 29,589 36.600 36.302 37.713 36.872 
AvGrate 27.646 26.131 28.034 27.270 30.718 29.842 30.472 30.344 


Relation with school composition variables. We computed the correlation of school 
ranks based on each school performance model with school composition variables to determine 
whether estimates were related to the aggregated student characteristics in each school. Table 17 
shows these correlations for mathematics and reading/language arts in the elementary school and 
middle school samples. Correlations of model estimates with school composition variables 
within each individual cohort are presented in Appendix F. The rightmost column of Table 17 
shows the correlation of each school performance model averaged over all of the school 
composition variables. As can be seen, correlations of the status models, PP and MLMO, ranged 
from -.153 to -.284 depending on content and grade level band and were noticeably higher than 
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the correlations of the other school performance models with school composition variables, 
which ranged from -.010 to +.181 depending on content and grade level band. 


Table 17 


Spearman's Correlations of School Ranks with School Composition Variables by Content Area 
and Grade Level Band 


Elementary School Mathematics 


Ethnic School 
Model EDS EL SWD Female Minority Size Mean 


PP -0.526 -0.323 = -0.205 0.001 -0.282 0.123 -0.202 
MLMO -0.616 -0.301 -0.184  -0.009 -0.272 =0.213—--0.195 
Gain -0.015 0.053 0.008 = -0.003 0.022 0.114 0.030 
TM -0.053 0.020 -0.011  -0.011 -0.012 0.102 0.006 
SGP -0.159 -0.013 -0.039 = -0.005 -0.014 0.172  -0.010 
VAM _ -0.183 -0.017_—_-0.053 0.013 -0.025 0.188 -0.013 
Grate -0.006 0.034 0.025 -0.011 -0.002 0.071 0.018 
AvGrate -0.029 0.086 -0.027 0.038 0.081 0.173 0.054 


Elementary School Reading/Language Arts 


Ethnic School 
Model EDS EL SWD Female Minority Size Mean 


PP -0.610 -0.491 -0.201 0.002 -0.426 0.040 -0.281 
MLMO_ -0.722 -0.480 -0.181  -0.004 -0.437 0.123 -0.284 
Gain 0.153 0.174 0.083 = -0.027 0.134 0.060 0.096 
TM -0.085 0.031 0.039  -0.026 0.018 0.089 0.011 
SGP -0.225 -0.062 -0.043 0.001 -0.083 0.112 -0.050 
VAM_ -0.264 -0.095 -0.049 0.006 -0.110 0.126 -0.064 
Grate 0.139 0.116 0.058 0.000 0.089 0.007 0.068 
AvGrate 0.224 0.273 0.060 0.005 0.202 0.054 0.136 


Middle School Mathematics 


Ethnic School 
Model EDS EL SWD Female Minority Size Mean 


PP -0.385 -0.211 -0.266 0.040 -0.213, 0.119 -0.153 
MLMO_ -0.488 -0.149 -0.178 0.005 -0.141 0.203 -0.125 
Gain -0.034 0.045 -0.013  -0.013 0.058 0.147 0.032 
T -0.094 0.008 -0.070  -0.010 0.004 0.158 -0.001 
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SGP -0.015 0.085 -0.016 0.000 0.081 0.173 0.051 
VAM -0.011 0.074 -0.023 0.002 0.074 0.168 0.047 
Grate -0.112 -0.008 -0.037  -0.020 0.017. 0.134 -0.004 
AvGrate 0.119 0.134 0.002 0.019 0.112 0.130 0.086 
Middle School Reading/Language Arts 
Ethnic School 
Model EDS EL SWD- Female Minority Size Mean 
PP -0.559 -0.385  -0.272 = 0.133 -0.401 0.053 = -0.239 
MLMO -0.687 -0.369 -0.241 0.108 -0.394 0.105  -0.246 
Gain 0.338 0.227 0.136 — -0.021 0.192 0.030 0.150 
TM -0.031 0.029 0.017 0.043 -0.002 0.137 0.032 
SGP -0.075 0.027 -0.021 0.051 -0.019 0.119 0.014 
VAM -0.055 0.034 = -0.019 ~=0.039 -0.030 0.103 0.012 
Grate 0.381 0.238 0.165 = -0.055 0.207 0.009 0.158 
AvGrate 0.398 0.311 0.132 -0.022 0.227 0.038 0.181 
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Relation of school ranks with SWD school composition. We also specifically 
examined the relations between the percentage of SWD students served by a school and the 
school ranks based on the school performance model. Table 18 shows these correlations for 
mathematics and reading/language arts in the elementary school and middle school samples 
averaged over cohorts. Correlations of model estimates with SWD school composition within 
each individual cohort are presented in Appendix G. As can be seen in the bottom row of Table 
18, on average, there was a substantially higher correlation of the status models (PP and MLMO) 
with school SWD composition than the other school performance models. With the PP and 
MLMO models, school ranks were higher with lower percentages of SWD students in the school 
and school ranks were lower as schools served larger proportions of SWD. Little relation was 
present between school ranks based on the other models and SWD school composition. 


Table 18 


Average School Rank as a Function of the Percentage of SWD in the School by Model, Content 


Area, and Grade Level Band 


Content Area and 


Grade Level Band PP MLMO ~~ Gain T™ SGP VAM Grate  AvGrate 
Math Elementary -0.205  -0.184 0.008 -0.011 -0.039 -0.053 0.025 -0.027 
Reading/Language -0.201 -0.181 0.083 0.039 -0.043 -0.049 0.058 0.060 
Arts Elementary 
Math Middle -0.266 -0.178 -0.013 -0.070 -0.016 -0.023  -0.037 0.002 
Reading/Language -0.272 -0.241 0.136 0.017 -0.021 -0.019 0.165 0.132 


Arts Middle 
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Mean -0.236 -0.196 0.054 -0.006 -0.030 -0.036 0.053 0.042 


Summary of Section B. We evaluated the school ranks arising from eight alternative 
models for estimating school academic performance in mathematics and reading/language arts 
across three sequential cohorts of students. As with the school performance estimates described 
in Section A, substantial variability in school ranks was present across the three student cohorts 
regardless of content area or grade level band. Using any of our comparison criteria (Spearman’s 
correlations, absolute difference in ranks, RMSD), there was somewhat less variability across 
cohorts for status models (PP and MLMO) than for the models that used more than one year of 
data. When we compared school ranks arising from one model to school ranks from other 
models, we found substantial disagreement across models. Generally, the PP and MLMO status 
models were more similar to each other and both differed from the remaining models that 
examined more than one year of data. Comparison of model estimates to school composition 
variables showed that the status models (PP and MLMO) had substantially higher correlations 
than the remaining school performance models. Finally, we correlated school ranks arising from 
the eight performance models with the percentage of SWD in each school. As with the school 
performance model estimates, we found that the status models were more highly correlated with 
SWD school composition but there was little relation of the other model estimates with the 
percentage of SWD students in the school. 


Conclusion 


This technical report described the Oregon results of a large study examining eight 
alternative methods of estimating school performance. We represented school performance in 
two ways, the actual model estimates and school ranks based on model estimates. In addition to 
an Overview of study methods and procedures and this Oregon report, there are additional 
reports describing results for the three other states (AZ, NC, PA) included in the study. 


A number of general conclusions can be drawn from the results of the Oregon analyses. 
First, model representations of school performance over successive cohorts of students were very 
unstable, irrespective of whether representations were based on school performance model 
estimates or on school ranks. There was somewhat greater stability for status models (PP, 
MLM0O) than for the multiyear models. Nonetheless, even with the most stable model, MLMO, 
Spearman’s correlations showed that less than two-thirds of the variance was common across 
cohorts, and over all the models, there was substantial instability over cohorts. These results 
were also reflected in the examination of differences in absolute or average (RMSD) differences 
in ranks over cohorts. 


Our examination of the relations of the school performance models with each other 
produced similar results. Generally, there was agreement between the two status model 
estimates (PP and MLMO) that were based on a single year of data, but these two models did not 
agree with the remaining multiyear models. However, there was some substantial agreement of 
the multiyear models with each other with some variations. In general, the AvGrate model 
showed the least agreement with the other multiyear models. 


We also examined the relation of school performance model estimates with variables 
describing the student composition of the schools. These results showed a pattern of results that 
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differed between the status and the multiyear models. The two status models had substantially 

higher correlations with school composition variables than the multiyear models. This was also 
true in terms of the percentage of SWD students served by a school. The higher the percentage 
of SWD in the school, the lower the status model estimates of school performance. 


Thus, the Oregon results showed consistent patterns of instability of estimates of school 
performance over successive cohorts of students, different estimates of school performance 
depending on the model chosen, especially for status versus multiyear models, and stronger 
relations of status models with the student composition of the school than multiyear models. 
Taken together, these results suggest the need for substantial caution in the way that school 
performance models are used and interpreted. Cohort instability suggests that rolling averages or 
some other mechanism is needed to provide more dependable depictions of school performance 
that are more stable over time. The substantial disagreement among the school performance 
models suggests that the choice of model matters a great deal. This choice should be made very 
carefully. A single model estimate of school performance may not be trustworthy and may need 
to be augmented by the results from additional models or metrics of school performance. 
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Appendix A 


Correlations among School Performance Model Estimates for Each Individual Cohort by 
Content Area and Grade Level Band. 


Mathematics Elementary Schools 
Cohort 1 


Model PP MLMO_ Gain TM SGP VAM Grate AvGrate 


PP 0.805 0.150 0.295 0.465 0.473 0.049 0.254 
MLMO 0.129 0.236 0.521 0.567 0.003 0.336 
Gain 0.887 0.816 0.832 0.952 0.515 
T™ 0.792 0.796 0.827 0.496 
SGP 0.956 0.659 0.756 
VAM 0.665 0.800 
Grate 0.253 
Cohort 2 
Model PP MLMO Gain TM SGP VAM Grate AvGrate 
PP 0.832 0.364 0.375 0.527 0.542 0.296 0.426 
MLMO 0.372 0.313 0.546 0.584 0.321 0.421 
Gain 0.901 0.892 0.916 0.970 0.691 
T™™ 0.826 0.826 0.865 0.640 
SGP 0.961 0.800 0.851 
VAM 0.822 0.890 
Grate 0.520 
Cohort 3 
Model PP MLMO Gain TM SGP VAM Grate AvGrate 
PP 0.842 0.268 0.405 0.494 0.534 0.161 0.469 
MLMO 0.328 0.400 0.583 0.625 0.214 0.520 
Gain 0.893 0.838 0.868 0.959 0.512 
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TM 0.817 0.820 0.838 0.511 
SGP 0.955 0.692 0.803 
VAM 0.719 0.840 
Grate 0.267 
AvGrate 
Mathematics Middle Schools 
Cohort 1 
Model PP MLMO Gain TM SGP VAM Grate AvGrate 
PP 0.770 0.360 0.530 0.435 0.421 0.306 0.287 
MLMO 0.383 0.389 0.460 0.520 0.342 O72 
Gain 0.845 0.839 0.878 0.932 0.500 
T™™ 0.719 0.705 0.781 0.366 
SGP 0.931 0.671 0.780 
VAM 0.723 0.833 
Grate 0.229 
Cohort 2 
Model PP MLMO Gain TM SGP VAM Grate AvGrate 
PP 0.796 0.425 0.523 0.535 0.512 0.393 0.354 
MLMO 0.378 0.409 0.481 0.500 0.388 0.287 
Gain 0.889 0.897 0.913 0.957 0.669 
T™ 0.844 0.844 0.840 0.640 
SGP 0.953 0.783 0.841 
VAM 0.818 0.886 
Grate 0.484 
Cohort 3 
Model PP MLMO Gain TM SGP VAM Grate AvGrate 
PP 0.790 0.583 0.662 0.568 0.591 0.594 0.502 
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MLMO 0.638 0.616 0.596 0.620 0.706 0.457 
Gain 0.927 0.904 0.936 0.965 0.759 
T™ 0.870 0.877 0.887 0.726 
SGP 0.962 0.832 0.901 
VAM 0.868 0.932 
Grate 0.630 
Reading/Language Arts Elementary Schools 
Cohort 1 
Model PP MLMO~ Gain TM SGP VAM_ Grate AvGrate 
PP 0.828 -0.012 0.276 0.466 0.547 -0.152  -0.099 
MLMO -0.184 0.208 0.455 0.522 -0.302 = -0.232 
Gain 0.718 0.651 0.670 0.907 0.504 
T™™ 0.722 0.721 0.593 0.345 
SGP 0.917 0.409 0.501 
VAM 0.435 0.529 
Grate 0.213 
Cohort 2 
Model PP MLMO~ Gain TM SGP VAM_ Grate AvGrate 
PP 0.838 0.071 0.200 0.432 0.497 -0.008 0.023 
MLMO -0.028 0.124 0.441 0.509 -0.102  -0.037 
Gain 0.769 0.727 0.766 0.931 0.534 
T™™ 0.673 0.684 0.685 0.456 
SGP 0.914 0.571 0.642 
VAM 0.605 0.696 
Grate 0.286 
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Cohort 3 
Model PP MLMO~ Gain TM SGP VAM_ Grate AvGrate 
PP 0.865 -0.104 0.303 0.445 0.512 0.122 = -0.055 
MLMO -0.108 0.254 0.438 0.502 0.044 = -0.139 
Gain -0.219 0.065 0.060 -0.512 0.626 
TM 0.732 0.743 0.738 0.418 
SGP 0.909 0.624 0.618 
VAM 0.689 0.662 
Grate 0.322 
Reading/Language Arts Middle Schools 
Cohort 1 
Model PP MLMO Gain TM SGP VAM_ Grate AvGrate 
PP 0.851 -0.197 0.383 0.390 0.347 -0.357 -0.192 
MLMO -0.352 0.227 0.324 0.291 -0.501 -0.321 
Gain 0.624 0.646 0.690 0.916 0.616 
T™ 0.730 0.715 0.470 0.351 
SGP 0.897 0.417 0.567 
VAM 0.509 0.685 
Grate 0.468 
Cohort 2 
Model PP MLMO Gain TM SGP VAM_ Grate AvGrate 
PP 0.827 -0.106 0.342 0.401 0.392  -0.303 -0.248 
MLMO -0.329 0.146 0.291 0.349 -0.462 -0.381 
Gain 0.706 0.647 0.649 0.891 0.549 
TM 0.683 0.657 0.517 0.277 
SGP 0.880 0.368 0.542 
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VAM 0.439 0.584 
Grate 0.354 
Cohort 3 
Model PP MLMO Gain TM SGP VAM_ Grate AvGrate 
PP 0.880 -0.113 0.357 0.393 0.457 -0.262 0.010 
MLMO -0.208 0.306 0.412 0.462 -0.351 -0.048 
Gain 0.705 0.689 0.686 0.942 0.642 
T™™ 0.783 0.786 0.592 0.524 
SGP 0.913 0.528 0.685 
VAM 0.549 0.776 
Grate 0.534 
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Appendix B 


Correlations of School Performance Model Estimates with School Composition Variables 
for each Individual Cohort by Content Area and Grade Level Band. 


Mathematics Elementary Schools 
Cohort 1 


Model EDS EL SWD Female Minority School Size 


PP -0.457 -0.333 -0.201  -0.017 -0.315 0.148 
MLMO_ -0.603 -0.348 -0.194 -0.058 -0.315 0.242 
Gain 0.102 0.156 0.081 -0.022 0.099 0.164 
TM 0.000 0.113 0.006 -0.015 0.043 0.174 
SGP -0.132 0.028 -0.033 -0.062 0.005 0.248 
VAM  -0.156 0.020 -0.029  -0.033 -0.001 0.270 
Grate 0.117 0.151 0.099 -0.009 0.076 0.110 
AvGrate 0.025 0.141 -0.014 -0.010 0.149 0.223 


Cohort 2 


Model EDS EL SWD Female Minority School Size 


PP -0.507 -0.319 -0.209 0.075 -0.291 0.181 
MLMO_ -0.636 -0.344 -0.223 0.025 -0.293 0.238 
Gain -0.039 0.080 0.019 0.023 0.027 0.093 
TM -0.001 0.108 -0.012 0.046 0.042 0.093 
SGP -0.122 0.026 -0.014 0.031 -0.001 0.142 
VAM -0.146 0.029 -0.041 0.045 0.000 0.170 
Grate -0.052 0.051 0.019 0.017 -0.005 0.063 
AvGrate 0.006 0.128 -0.011 0.058 0.106 0.166 


Cohort 3 


Model EDS EL SWD Female Minority School Size 


PP -0.536 -0.334 -0.172  -0.021 -0.351 0.111 
MLMO_ -0.638 -0.327 -0.167  -0.004 -0.288 0.203 
Gain -0.109 -0.080 -0.049  -0.009 -0.064 0.039 
TM -0.173 -0.126 -0.041 -0.049 -0.119 0.025 
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SGP -0.219 -0.102 -0.060 0.003 -0.088 0.122 
VAM -0.239 = -0.120 -0.086 0.005 -0.104 0.120 
Grate -0.104 -0.105 -0.031 -0.028 -0.080 0.013 

AvGrate -0.092 0.000 -0.058 0.033 -0.007 0.121 

Mathematics Middle Schools 

Cohort | 
Model EDS EL SWD Female Minority School Size 

PP -0.362 -0.198 -0.248 -0.013 -0.309 0.166 
MLMO -0.461 -0.189 -0.169 -0.078 -0.204 0.212 
Gain -0.050 0.116 -0.016  -0.021 0.112 0.156 
TM -0.076 0.117 -0.132 0.011 0.024 0.227 

SGP -0.005 0.130 0.039 — -0.009 0.103 0.165 
VAM -0.031 0.118 0.024  -0.038 0.089 0.188 
Grate -0.109 0.055 -0.038 -0.054 0.068 0.135 

AvGrate 0.101 0.154 0.089  -0.006 0.094 0.144 
Cohort 2 
Model EDS EL SWD Female Minority School Size 
PP -0.331 -0.234 -0.275 0.084 -0.313 0.137 
MLMO_ -0.472 -0.258 -0.180 0.096 -0.210 0.221 
Gain 0.039 -0.018 -0.058 0.032 -0.040 0.065 
TM -0.015 -0.055 -0.099 0.035 -0.084 0.090 

SGP 0.011 0.008 -0.084 0.015 -0.046 0.128 
VAM 0.024 0.023 -0.050 0.026 -0.006 0.143 
Grate -0.025 -0.082 -0.059 0.029 -0.081 0.049 

AvGrate 0.166 0.157 0.009  -0.003 0.096 0.142 
Cohort 3 
Model EDS EL SWD Female Minority School Size 
PP -0.296 -0.254 -0.278 0.082 -0.175 0.205 
MLMO_ -0.464 -0.249 -0.187 0.010 -0.137 0.275 
Gain -0.016 -0.047 -0.137 0.080 0.097 0.198 
TM -0.049 -0.064 -0.124 0.065 0.076 0.181 

SGP 0.008 -0.003 -0.102 0.073 0.121 0.223 

VAM 0.024 -0.005 -0.118 0.073 0.104 0.211 
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Grate -0.137 -0.107 -0.126 0.057 0.009 0.229 
AvGrate 0.133 0.072 -0.096 0.070 0.151 0.165 
Reading/Language Arts Elementary Schools 
Cohort 1 

Model EDS EL SWD Female Minority School Size 
PP -0.539 -0.530 -0.174  -0.035 -0.488 0.073 
MLMO_ -0.671 -0.511 -0.187  -0.025 -0.464 0.153 
Gain 0.212 0.201 0.130 0.001 0.122 0.086 
TM -0.080 0.049 0.081  -0.033 -0.012 0.156 
SGP -0.187 -0.055 -0.021 0.001 -0.111 0.174 

VAM -0.265 -0.124 -0.034 -0.007 -0.176 0.187 

Grate 0.251 0.198 0.136 0.023 0.123 0.061 
AvGrate 0.253 0.320 0.065 0.006 0.232 0.044 
Cohort 2 

Model EDS EL SWD Female Minority School Size 
PP -0.610 -0.506 -0.255 0.013 -0.472 0.078 
MLMO_ -0.747 -0.530 -0.240 =0.013 -0.475 0.145 
Gain 0.100 0.175 0.019 -0.002 0.110 0.054 
TM -0.045 0.101 0.025  -0.005 0.046 0.084 
SGP -0.227 -0.051 -0.083  -0.008 -0.097 0.127 

VAM_ -0.271 -0.077 -0.106 0.005 -0.116 0.151 

Grate 0.119 0.152 0.035 0.000 0.101 0.004 
AvGrate 0.184 0.272 0.026  -0.002 0.190 0.105 
Cohort 3 

Model EDS EL SWD Female Minority School Size 
PP -0.651 -0.542 -0.184 0.029 -0.516 0.040 
MLMO_ -0.757  -0.537 -0.139 = 0.015 -0.492 0.099 
Gain 0.163 0.175 0.079 -0.071 0.120 0.060 
TM -0.091 0.004 0.000 0.046 -0.010 0.011 
SGP -0.215 -0.078 -0.019  -0.002 -0.108 0.029 

VAM_ -0.239 -0.129 -0.037 0.027 -0.130 0.038 

Grate 0.057 0.041 -0.009 0.046 0.047 -0.050 
AvGrate 0.275 0.264 0.080 -0.010 0.209 0.008 
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Reading/Language Arts Middle Schools 


Cohort 1 
Model EDS EL SWD Female Minority School Size 
PP -0.505 -0.412 -0.342 0.197 -0.496 0.104 
MLMO_ -0.628 -0.440 -0.253 0.109 -0.452 0.113 
Gain 0.312 0.259 0.127 = -0.006 0.197 0.017 
TM -0.005 0.031 -0.032 0.086 -0.082 0.103 
SGP -0.054 0.054 -0.041 0.065 -0.097 0.129 
VAM -0.017 0.055 -0.042 0.013 -0.058 0.138 
Grate 0.369 0.290 0.163  -0.061 0.236 -0.005 
AvGrate 0.412 0.360 0.102 -0.067 0.249 0.074 
Cohort 2 
Model EDS EL SWD Female Minority School Size 
PP -0.501 -0.451 -0.197 0.135 -0.510 0.031 
MLMO_ -0.668 -0.498 -0.192 0.162 -0.483 0.079 
Gain 0.359 0.280 0.083 = -0.053 0.187 -0.023 
TM 0.032 0.050 0.063 0.014 -0.032 0.042 
SGP -0.007 0.033 0.033 0.008 -0.036 0.093 
VAM_ -0.033 -0.002 -0.023 0.055 -0.087 0.064 
Grate 0.369 0.257 0.102 -0.057 0.178 -0.058 
AvGrate 0.483 0.403 0.133  -0.082 0.331 0.039 
Cohort 3 
Model EDS EL SWD Female Minority School Size 
PP -0.569 -0.468 -0.251 0.072 -0.438 0.129 
MLMO_ -0.693 -0.439 -0.205 0.087 -0.433 0.134 
Gain 0.306 0.175 0.065 -0.018 0.294 0.099 
TM -0.090 -0.078 -0.033 0.007 0.031 0.191 
SGP -0.128 -0.038 -0.097 0.096 -0.009 0.163 
VAM -0.124 -0.048 -0.048 0.053 -0.028 0.162 
Grate 0.357 0.219 0.113 -0.044 0.292 0.085 
AvGrate 0.287 0.254 0.060 0.033 0.195 0.058 
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Appendix C 


Correlations of School Performance Model Estimates with School Percentage SWD for 
each Individual Cohort by Content Area and Grade Level Band. 


Mathematics Elementary Schools 


Cohort PP MLMO Gain ™ SGP VAM_ Grate AvGrate 


1 -0.201 -0.194 0.081 0.006 -0.033 -0.029 0.099 ~~ -0.014 
2 -0.209 -0.223 0.019 -0.012 -0.014 -0.041 0.019  -0.011 
3 -0.172 -0.167 -0.049 -0.041 -0.060 -0.086 -0.031 -0.058 


Mathematics Middle Schools 


Cohort PP MLMO Gain ™ SGP VAM_ Grate AvGrate 


1 -0.248 -0.169 -0.016 -0.132 0.039 0.024 -0.038 0.089 
2 -0.275 -0.180 -0.058 -0.099 -0.084 -0.050 -0.059 0.009 
3 -0.278 =-0.187  -0.137 -0.124 -0.102 -0.118 -0.126 -0.096 


Reading/Language Arts Elementary Schools 


Cohort PP MLMO- Gain ™ SGP VAM Grate AvGrate 


1 -0.174 -0.187 0.130 0.081 -0.021 -0.034 0.136 0.065 
2 -0.255 -0.240 0.019 0.025 -0.083 -0.106 0.035 0.026 
3 -0.184 -0.139 0.079 0.000 -0.019 -0.037 -0.009 0.080 


Reading/Language Arts Middle Schools 


Cohort PP MLMO_ Gain ™ SGP VAM _ Grate AvGrate 


1 -0.342 -0.253 0.127 -0.032 -0.041 -0.042 0.163 0.102 
2 -0.197 -0.192 0.083 0.063 0.033 -0.023 0.102 0.133 
3 -0.251 = -0.205 0.065 -0.033 -0.097 -0.048 0.113 0.060 
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Appendix D 


Proportion of Elementary or Middle Schools within 5, 10, or 20 Ranks of Each Other for each 
Pair of School Performance Models in Mathematics and Reading/Language Arts by Cohort. 


Cohort | Cohort 2 Cohort 3 
Model 
Comparison r=5 r=10 r=20 r=5 r=10 r=20 r=5 r=10 r=20 
PP vs. MLMO 


Math Elementary 0.315 0.536 0.835 0.401 0.594 0.838 0.373 0.606 0.832 


Reading/Language 0.366 0.563 0.823 0.341 0.581 0.834 0.404 0.609 0.863 
Arts Elementary 


Math Middle 0.370 0.560 0.819 0.342 0.605 0.844 0.350 0.514 0.802 


Reading/Language 0.428 0.663 0.881 0.387 0.617 0.848 0.403 0.654 0.897 
Arts Middle 


Mean 0.370 0.580 0.839 0.368 0.599 0.841 0.382 0.596 0.848 


PP vs. Gain 
Math Elementary 0.149 0.236 0.421 0.150 0.276 0.472 0.144 0.243 0.431 


Reading/Language 0.109 0.199 0.359 0.121 0.209 0.371 0.101 0.204 0.338 
Arts Elementary 


Math Middle 0.119 0.243 0.428 0.206 0.333 0.490 0.210 0.342 0.584 


Reading/Language 0.107 0.177 0.342 0.099 0.181 0.317 0.086 0.181 0.321 
Arts Middle 


Mean 0.121 0.214 0.388 0.144 0.250 0412 0.135 0.242 0.418 


PP vs. TM 

Math Elementary 0.160 0.257 0.469 0.165 0.285 0.470 0.155 0.284 0.474 
Reading/Language 0.132 0.257 0.469 0.129 0.215 0.427 0.139 0.260 0.470 
Arts Elementary 


51 


52 


Math Middle 0.169 0.317 0.551 0.210 0.379 0.580 0.272 0.374 0.617 
Reading/Language 0.144 0.300 0.481 0.181 0.276 0.490 0.210 0.317 0.523 
Arts Middle 

Mean 0.151 0.283 0.492 0.171 0.289 0.492 0.194 0.309 0.521 
PP vs. SGP 

Math Elementary 0.178 0.305 0.525 0.205 0.309 0.548 0.168 0.302 0.561 
Reading/Language 0.180 0.315 0.538 0.137 0.275 0.492 0.189 0.338 0.543 
Arts Elementary 

Math Middle 0.177 0.296 0.502 0.214 0.362 0.551 0.214 0.358 0.568 
Reading/Language 0.202 0.292 0.506 0.140 0.280 0.465 0.160 0.276 0.527 
Arts Middle 

Mean 0.184 0.302 0.518 0.174 0.306 0.514 0.183 0.318 0.550 
PP vs. VAM 

Math Elementary 0.186 0.318 0.540 0.186 0.318 0.554 0.175 0.323 0.569 
Reading/Language 0.194 0.344 0.588 0.171 0.291 0.520 0.215 0.359 0.566 
Arts Elementary 

Math Middle 0.173 0.28 0.486 0.214 0.354 0.535 0.193 0.342 0.580 
Reading/Language 0.160 0.288 0.481 0.128 0.23 0.510 0.160 0.292 0.494 
Arts Middle 

Mean 0.178 0.308 0.524 0.175 0.298 0.530 0.186 0.329 0.552 
PP vs. Grate 

Math Elementary 0.132 0.231 0.388 0.137 0.256 0.460 0.124 0.223 0.396 
Reading/Language 0.089 0.174 0.333 0.096 0.199 0.351 0.127 0.232 0.399 
Arts Elementary 

Math Middle 0.107 0.226 0.449 0.181 0.305 0.473 0.243 0.395 0.593 
Reading/Language 0.070 0.119 0.263 0.103 0.165 0.280 0.070 0.144 0.325 
Arts Middle 
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Mean 0.100 0.188 0.358 0.129 0.231 0.391 0.141 0.248 0.428 
PP vs. AvGrate 

Math Elementary 0.167 0.269 0.485 0.153 0.285 0.518 0.153 0.289 0.503 
Reading/Language 0.086 0.180 0.373 0.114 0.202 0.348 0.099 0.182 0.349 
Arts Elementary 

Math Middle 0.152 0.288 0.453 0.189 0.321 0.481 0.198 0.313 0.527 
Reading/Language 0.082 0.193 0.296 0.103 0.169 0.296 0.099 0.222 0.383 
Arts Middle 

Mean 0.122 0.232 0.402 0.140 0.244 0.411 0.137 0.252 0.440 
MLM0O vs. Gain 

Math Elementary 0.119 0.243 0.436 0.162 0.287 0.482 0.153 0.257 0.472 
Reading/Language 0.083 0.161 0.321 0.099 0.169 0.343 0.113 0.227 0.381 
Arts Elementary 

Math Middle 0.152 0.284 0.457 0.144 0.309 0.502 0.218 0.366 0.621 
Reading/Language 0.095 0.169 0.329 0.078 0.169 0.288 0.099 0.173 0.350 
Arts Middle 

Mean 0.112 0.214 0.386 0.121, “0.233 0.404 0.146 0.256 0.456 
MLM0 vs. TM 

Math Elementary 0.145 0.269 0.480 0.137 0.269 0.474 0.150 0.281 0.487 
Reading/Language 0.134 0.238 0.406 O12) 0.222 0.394 0.141 0.250 0.462 
Arts Elementary 

Math Middle O52 O37" 0,523 0.123 0.267 0.506 0.226 0.374 0.613 
Reading/Language 0.107 0.235 0.416 0.152 0.251 0.420 0.193 0.317 0.506 
Arts Middle 

Mean 0.134 0.265 0.456 135. . (252 0.448 0.178 0.306 0.517 
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MLM0O vs. SGP 

Math Elementary 0.188 0.342 0.569 0.191 0.343 0.571 0.186 0.350 0.587 
Reading/Language 0.184 0.311 0.538 0.164 0.278 0.464 0.169 0.291 0.525 
Arts Elementary 

Math Middle 0.165 0.300 0.527 0.173 0.337 0.560 0.23 0.379 0.576 
Reading/Language 0.156 0.272 0.519 0.148 0.259 0.449 0.193 0.280 0.556 
Arts Middle 

Mean 0.173 0.306 0.538 0.169 0.304 0.511 0.194 0.325 0.561 
MLMO vs. VAM 

Math Elementary 0.219 0.353 0.584 0.188 0.340 0.589 0.213 0.365 0.612 
Reading/Language 0.177 0.325 0.579 0.171 0.295 0.520 0.177 0.328 0.548 
Arts Elementary 

Math Middle 0.165 0.313 0.543 0.189 0.325 0.560 0.235 0.374 0.593 
Reading/Language 0.152 0.251 0.498 0.111 0.267 0.514 0.160 0.317 0.531 
Arts Middle 

Mean 0.178 0.310 0.551 0.165 0.307 0.546 0.196 0.346 0.571 
MLM6O vs. Grate 

Math Elementary 0.116 0.210 0.381 0.135 0.284 0.472 0.120 0.229 0.408 
Reading/Language 0.075 0.149 0.298 0.089 0.180 0.315 0.099 0.214 0.397 
Arts Elementary 

Math Middle 0.160 0.284 0.457 0.156 0.292 0.506 0.292 0.449 0.691 
Reading/Language 0.058 0.119 0.272 0.078 0.165 0.280 0.111 0.173 0.313 
Arts Middle 

Mean 0.102 0.190 0.352 0.114 0.230 0.393 0.156 0.266 0.452 
MLM0O vs. 

AvGrate 

Math Elementary 0.198 0.312 0.510 0.167 0.309 0.528 0.178 0.297 0.530 
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Reading/Language 0.086 0.167 0.341 0.096 0.187 0.358 0.114 0.194 0.349 
Arts Elementary 

Math Middle 0.193 0.333 0.502 0.156 0.280 0.457 0.193 0.313 0.490 
Reading/Language 0.095 0.165 0.300 0.099 0.173 0.296 0.119 0.210 0.399 
Arts Middle 

Mean 0.143 0.244 0.413 0.130 0.237 0.410 0.151 0.254 0.442 
Gain vs. TM 

Math Elementary 0.403 0.614 0.866 0.404 0.642 0.884 0.406 0.639 0.875 
Reading/Language 0.272 0.439 0.697 0.281 0.464 0.738 0.091 0.180 0.329 
Arts Elementary 

Math Middle 0.395 0.597 0.831 0.387 0.613 0.848 0.481 0.700 0.918 
Reading/Language 0.226 0.374 0.588 0.202 0.412 0.650 0.267 0.461 0.634 
Arts Middle 

Mean 0.324 0.506 0.745 0.318 0.533 0.780 0.311 0.495 0.689 
Gain vs. SGP 

Math Elementary 0.315 0.490 = 0.777 0.429 0.639 0.865 0.327 0.526 0.795 
Reading/Language 0.250 0.401 0.594 0.228 0.389 0.647 0.141 0.233 0.416 
Arts Elementary 

Math Middle 0.329 0.560 0.798 0.383 0.613 0.844 0.469 0.675 0.897 
Reading/Language 0.226 0.370 0.642 0.247 0.391 0.601 0.280 0.387 0.646 
Arts Middle 

Mean 0.280 0.455 0.703 0.322 0.508 0.739 0.304 0.455 0.688 
Gain vs. VAM 

Math Elementary 0.292 0.523 0.795 0.436 0.670 0.893 0.340 0.561 0.818 
Reading/Language 0.220 0.376 0.616 0.243 0.399 0.666 0.113 0.228 0.417 
Arts Elementary 

Math Middle 0.395 0.630 0.848 0.416 0.609 0.868 0.506 0.708 0.934 
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Reading/Language 0.206 0.346 0.650 0.206 0.366 0.638 0.255 0.391 0.658 
Arts Middle 

Mean 0.278 0.469 0.727 0.325 0.511 0.766 0.304 0.472 0.707 
Gain vs. Grate 

Math Elementary 0.512 0.795 0.979 0.660 0.866 0.987 0.578 0.807 0.974 
Reading/Language 0.424 0.689 0.912 0.500 0.743 0.960 0.079 0.139 0.272 
Arts Elementary 

Math Middle 0.498 0.765 0.942 0.551 0.798 0.979 0.609 0.885 0.992 
Reading/Language 0.424 0.704 0.938 0.436 0.704 0.901 0.543 0.811 0.975 
Arts Middle 

Mean 0.464 0.738 0.943 0.537 0.778 0.957 0.452 0.661 0.803 
Gain vs. AvGrate 

Math Elementary 0.200 0.351 0.558 0.241 0.432 0.677 0.193 0.328 0.556 
Reading/Language 0.204 0.353 0.545 0.175 0.318 0.551 0.215 0.404 0.616 
Arts Elementary 

Math Middle 0.226 0.366 0.556 0.272 0.362 0.617 0.296 0.469 0.704 
Reading/Language 0.222 0.317 0.605 0.214 0.374 0.588 0.276 0.428 0.658 
Arts Middle 

Mean 0.213 0.347 0.566 0.226 0.372 0.608 0.245 0.407 0.634 
T vs. SGP 

Math Elementary 0.315 0.485 0.774 0.363 0.533 0.789 0.309 0.517 0.784 
Reading/Language 0.240 0.424 0.639 0.247 0.394 0.619 0.281 0.452 0.697 
Arts Elementary 

Math Middle 0.235 0.444 0.712 0.379 0.584 0.782 0.391 0.572 0.835 
Reading/Language 0.337 0.502 0.745 0.272 0.399 0.667 0.333 0.490 0.737 
Arts Middle 

Mean 0.282 0.464 0.718 0.315 0.478 0.714 0.328 0.508 0.763 
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TM vs. VAM 

Math Elementary 0.300 0.492 0.769 0.351 0.554 0.799 0.330 0.513 0.781 
Reading/Language 0.232 0.387 0.647 0.233 0.386 0.651 0.273 0.430 0.689 
Arts Elementary 

Math Middle 0.263 0.465 0.737 0.350 0.576 0.798 0.387 0.572 0.840 
Reading/Language 0.259 0.449 = 0.733 0.235 0.383 0.675 0.317 0.498 0.774 
Arts Middle 

Mean 0.264 0.448 0.722 0.292 0.475 0.731 0.327 0.503 0.771 
TM vs. Grate 

Math Elementary 0.317 0.526 0.792 0.360 0.554 0.838 0.335 0.551 0.790 
Reading/Language 0.215 0.373 0.613 0.212 0.396 0.671 0.257 0.422 0.684 
Arts Elementary 

Math Middle 0.309 0.547 0.790 0.337 0.523 0.802 0.457 0.683 0.872 
Reading/Language 0.173 0.313 0.514 0.177 0.362 0.543 0.259 0.366 0.601 
Arts Middle 

Mean 0.254 0.440 0.677 0.272 0.459 0.714 0.327 0.506 0.737 
TM vs. AvGrate 

Math Elementary 0.185 0.345 0.543 0.233 0.394 0.640 0.186 0.328 0.543 
Reading/Language 0.142 0.288 0.459 0.179 0.298 0.526 0.177 0.290 0.475 
Arts Elementary 

Math Middle 0.218 0.333 0.498 0.218 0.403 0.626 0.239 0.412 0.667 
Reading/Language 0.160 0.272 0.514 0.123 0.276 0.444 0.206 0.346 0.535 
Arts Middle 

Mean 0.176 0.310 0.504 0.188 0.343 0.559 0.202 0.344 0.555 
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SGP vs. VAM 

Math Elementary 0.583 0.830 0.972 0.579 0.828 0.969 0.564 0.807 0.980 
Reading/Language 0.417 0.671 0.921 0.444 0.685 0.897 0.429 0.682 0.904 
Arts Elementary 

Math Middle 0.564 0.794 0.959 0.609 0.856 0.984 0.621 0.848 0.979 
Reading/Language 0.424 0.671 0.926 0.420 0.700 0.905 0.432 0.728 0.930 
Arts Middle 

Mean 0.497 0.742 0.944 0.513 0.767 0.939 0.511 0.766 0.948 
SGP vs. Grate 

Math Elementary 0.221 0.381 0.634 0.322 0.528 0.748 0.224 0.408 0.653 
Reading/Language 0.164 0.293 0.512 0.152 0.315 0.540 0.247 0.379 0.604 
Arts Elementary 

Math Middle 0.243 0.379 0.691 0.317 0.457 0.695 0.387 0.556 0.782 
Reading/Language 0.177 0.292 0.502 0.144 0.276 0.490 0.210 0.325 0.527 
Arts Middle 

Mean 0.201 0.336 0.585 0.234 0.394 0.618 0.267 0.417 0.642 
SGP vs. AvGrate 

Math Elementary 0.282 0.487 0.731 0.337 0.589 0.818 0.333 0.502 0.761 
Reading/Language 0.185 0.316 0.543 0.224 0.374 0.593 0.238 0.389 0.611 
Arts Elementary 

Math Middle 0.296 0.477 = 0.745 0.329 0.560 0.802 0.387 0.630 0.872 
Reading/Language 0.210 0.362 0.597 0.202 0.354 0.613 0.255 0.391 0.634 
Arts Middle 

Mean 0.243 0.410 0.654 0.273 0.469 0.706 0.303 0.478 0.720 
VAM vs. Grate 

Math Elementary 0.215 0.373 0.624 0.332 0.531 0.771 0.254 0.416 0.672 
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Reading/Language 0.172 0.316 0.512 0.209 0.353 0.581 0.248 0.396 0.627 
Arts Elementary 

Math Middle 0.263 0.440 0.712 0.296 0.469 0.708 0.374 0.580 0.815 
Reading/Language 0.177 0.325 0.514 0.193 0.321 0.523 0.173 0.346 0.580 
Arts Middle 

Mean 0.207 0.364 0.590 0.258 0.418 0.646 0.262 0.434 0.674 
Grate vs. AvGrate 

Math Elementary 0.150 0.267 0.459 0.219 0.338 0.586 0.137 0.257 0.459 
Reading/Language 0.136 0.253 0.430 0.139 0.260 0.434 0.139 0.255 0.449 
Arts Elementary 

Math Middle 0.169 0.296 0.461 0.214 0.300 0.490 0.230 0.391 0.601 
Reading/Language 0.202 0.333 0.568 0.181 0.288 0.486 0.189 0.391 0.601 
Arts Middle 

Mean 0.164 0.287 0.480 0.188 0.296 0.499 0.174 0.324 0.528 
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Appendix E 


RMSD in School Ranks for Pairs of School Performance Models for each Individual 
Cohort by Content Area and Grade Level Band. 


Elementary School Mathematics: Cohort 1 


Model MLMO Gain ™ SGP VAM Grate AvGrate 


PP 15.770 37.362 34.105 29.798 29.045 39.441 34.357 


MLMO 37.581 34.712 28.086 27.087 40.005 32.912 
Gain 13.586 17.328 16.493 8491 28.533 
T™ 18.316 17.633 16.537 29.001 
SGP 8.358 23.251 20.655 
VAM 22.979 19.137 
Grate 34.628 


Elementary School Mathematics: Cohort 2 


Model MLMO Gain ™ SGP VAM Grate AvGrate 


PP 14.395 32.726 33.384 28.644 27.823 34.156 31.510 


MLMO 31.685 33.452 27.287 26.152 33.020 30.914 
Gain 12.974 13.846 12.127 6.928 23.014 
T™ 17.736 17.345 14.738 24.819 
SGP 8.376 18.687 16.332 
VAM 17.860 14.033 
Grate 28.523 


Elementary School Mathematics: Cohort 3 


Model MLMO Gain ™ SGP VAM Grate AvGrate 


PP 14.438 34.826 31.582 29.163 27.95 37.177 30.428 


MLMO 33.463 31.036 26.362 25.182 36.223 28.276 
Gain 13.221 16.738 15.247 8.409 29.121 
TM 17.213 16.842 16.581 28.739 
SGP 8.232 23.134 18.582 
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VAM 22.268 17.376 
Grate 35.168 
Elementary School Reading/Language Arts: Cohort 1 

Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 15.540 41.213 34.408 29.400 27.321 43.555 42.161 
MLMO 43.497 35.963 29.225 27.463 45.584 43.963 
Gain 22.777 25.344 24.121 11.436 29.191 
T™ 22.777 22.356 25.866 33.432 
SGP 11.359 31.683 28.953 
VAM 30.928 28.344 
Grate 36.431 
Elementary School Reading/Language Arts: Cohort 2 

Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 15.237 39.573 36.531 31.045 29.042 40.759 40.435 
MLMO 41.291 38.112 30.641 28.763 42.498 40.938 
Gain 20.322 21.984 20.109 9.431 28.105 
T™ 23.757 23.045 22.685 30.209 
SGP 12.271 26.996 24.897 
VAM 25.955 23.274 
Grate 34.614 
Elementary School Reading/Language Arts: Cohort 3 

Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 13.361 42.724 33.727 30.142 28.365 38.305 41.723 
MLMO 42.277 34.736 30.326 28.634 39.893 42.711 
Gain 44.548 39.077 38.925 49.566 25.217 
T™ 21.761 21.312 21.682 31.506 
SGP 11.974 25.923 25.274 
VAM 24.214 23.890 
Grate 33.782 
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Middle School Mathematics: Cohort 1 


Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 15.945 33.532 29.275 31.237 31.078 33.991 34.467 
MLMO 31.716 30.267 28.905 28.418 32.702 32.596 
Gain 15.570 17.226 15.180 9.810 29.276 
T™ 21.650 20.802 16.673 31.588 
SGP 9.639 24.035 18.819 
VAM 22.930 16.925 
Grate 35.983 
Middle School Mathematics: Cohort 2 

Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 15.635 30.273 27.086 27.474 27.691 30.347 33.333 
MLMO 30.719 29.502 27.656 27.321 30.368 33.654 
Gain 15.684 13.993 12.869 8.252 24.499 
T™ 17.324 17.193 17.646 25.730 
SGP 7.731 19.772 16.862 
VAM 18.960 14.753 
Grate 30.439 
Middle School Mathematics: Cohort 3 

Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 17.076 26.629 24.410 27.029 26.004 25.949 28.927 
MLMO 23.594 23.467 24.903 24.606 21.557 29.608 
Gain 12.116 12.830 10.747 6.916 20.965 
T™ 15.701 14.937 13.012 22.505 
SGP 7.620 16.842 13.090 
VAM 15.846 11.246 
Grate 25.664 
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Middle School Reading/Language Arts: Cohort | 


Model MLMO Gain TM SGP VAM Grate AvGrate 
PP 13.241 43.882 32.089 32.422 33.161 46.984 44.585 
MLMO 46.167 35.268 33.305 34.104 49.012 45.698 
Gain 24.977 24.299 22.190 11.217 25.328 
TM 20.479 20.045 29.293 31.687 
SGP 11.441 30.771 26.530 
VAM 29.010 23.190 
Grate 30.048 
Middle School Reading/Language Arts: Cohort 2 

Model MLMO_ Gain ™ SGP VAM _ Grate AvGrate 
PP 15.000 44.032 34.833 32.958 32.319 46.304 45.287 
MLMO 46.575 37.457 34.303 33.357 48.333 47.229 
Gain 23.159 25.486 24.178 11.342 26.658 
T™ 24.126 22.259 26.615 33.678 
SGP 11.744 32.150 26.751 
VAM 30.725 26.232 
Grate 32.473 
Middle School Reading/Language Arts: Cohort 3 

Model MLMO_ Gain ™ SGP VAM _ Grate AvGrate 
PP 12.169 42.661 32.093 30.448 29.671 45.404 40.724 
MLMO 43.755 32.510 29.891 29.391 46.377 41.262 
Gain 23.650 23.746 22.771 8.753 24.015 
TM 19.919 18.565 26.819 28.573 
SGP 10.938 28.795 23.796 
VAM 28.036 20.678 
Grate 21129 
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Appendix F 


Correlations of School Ranks with School Composition Variables by Content Area and 
Grade Level Band for each Individual Cohort. 


Elementary School Mathematics: Cohort 1 


Ethnic School 
Model EDS EL SWD_ Female Minority Size Mean 


PP -0.490 -0.351 = -0.212) --0.025.—-0.273 — (0.124 ~—s -0.204 
MLMO -0.599  -0.325. -0.185 -0.045 = -0.282. 0.224 — -0.202 
Gain 0.105 0.164 0.066 -0.022 0.098 0.181 0.099 
TM -0.001 0.107 0.004 -0.033 0.042 0.185 0.051 
SGP -0.135 0.030 -0.025 -0.044 0.017 0.248 0.015 
VAM -0.153 0.029 -0.029 -0.023 0.008 0.265 0.016 
Grate 0.126 0.154 0.089 -0.018 0.076 0.132 0.093 


AvGrate 0.013 0.139 -0.008 0.001 0.134 0.223 0.084 


Elementary School Mathematics: Cohort 2 


Ethnic School 


Model EDS EL SWD_ Female Minority Size | Mean 
PP -0.548 = -0.338 = --0.213, 0.045) -0.270 = 0.142_~—s -0.197 
MLMO -0.626 -0.315 -0.198 0.033 -0.271 0.219 = -0.193 
Gain -0.042 0.054 0.015 0.036 0.033 0.116 0.035 
TM 0.002 0.056 0.003 0.047 0.029 0.088 0.038 
SGP -0.122 0.011 -0.014 0.035 0.016 0.144 0.012 
VAM -0.147 0.010 -0.034 0.059 0.015 0.168 0.012 
Grate -0.057 0.022 0.017 0.024 -0.003 0.076 0.013 


AvGrate 0.003 0.104 -0.007 0.073 0.103 0.158 0.072 
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Elementary School Mathematics: Cohort 3 


Ethnic School 
Model EDS EL SWD_ Female Minority Size Mean 
PP -0.539 -0.280 -0.191 -0.018 -0.303 0.103 = -0.205 
MLMO -0.622 -0.262 -0.169 -0.016 -0.262 0.195  -0.189 
Gain -0.109 -0.058 -0.057 -0.021 -0.067 0.046 = -0.044 
TM -0.160 -0.104 -0.039 -0.046 -0.108 0.034  -0.070 
SGP -0.220 -0.079 -0.077 -0.007 -0.074 0.123  -0.056 
VAM -0.248 -0.090 -0.097 0.004 -0.097 0.129 -0.066 
Grate -0.087 -0.073 -0.031 -0.038 -0.079 0.005 = -0.050 
AvGrate -0.104 0.015 -0.066 0.039 0.005 0.139 0.005 
Elementary School Reading/Language Arts: Cohort 1 
Ethnic School 
Model EDS EL SWD_ Female Minority Size Mean 
PP -0.555  -0.510 -0.181 -0.034 -0.424 0.028 = -0.279 
MLMO -0.681 -0.475 -0.178 -0.029 -0418 0.136 = -0.274 
Gain 0.222 0.183 0.120 -0.029 0.154 0.091 0.124 
TM -0.092 0.030 0.076 -0.085 0.043 0.156 0.021 
SGP -0.224 -0.069 -0.032 0.005 -0.079 0.183  -0.036 
VAM -0.279 =-0.120 -0.035 -0.016 -0.119 0.195  -0.062 
Grate 0.252 0.157 0.146 -0.017 0.131 0.052. 0.120 
AvGrate 0.234 0.311 0.055 -0.002 0.233 0.053. 0.147 
Elementary School Reading/Language Arts: Cohort 2 
Ethnic School 
Model EDS EL SWD_ Female Minority Size Mean 
PP -0.626 -0.488 -0.248 0.012 -0.414 0.052 = -0.285 
MLMO -0.741 -0.495 -0.225 0.015 -0.440 0.134 = -0.292 
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Gain 0.097. 0.180 0.041 -0.013 0.124 0.036 0.078 
TM -0.056 0.082 0.032 -0.004 0.059 0.085 0.033 
SGP -0.233 -0.051 -0.079 -0.001 -0.066 0.108 -0.054 
VAM -0.268 -0.066 -0.083 0.012 -0.089 0.123 = -0.062 
Grate 0.098 0.138 0.048 -0.013 0.094 -0.003 0.060 
AvGrate 0.187 0.266 0.045 0.013 0.184 0.093 0.131 
Elementary School Reading/Language Arts: Cohort 3 
Ethnic School 

Model EDS EL SWD_ Female Minority Size Mean 
PP -0.648 -0.476 -0.173 0.029 -0.440 0.041 = -0.278 
MLMO -0.745 -0.470 -0.141 0.002 -0.452 0.097 = -0.285 
Gain 0.141 0.159 0.088 -0.040 0.123 0.054 0.088 
TM -0.107 -0.019 0.008 0.010 -0.049 0.026 = -0.022 
SGP -0.218 -0.066 -0.019 -0.002 -0.102 0.047 = -0.060 
VAM -0.246 -0.099 -0.028 0.022 -0.121 0.061  -0.068 
Grate 0.066 0.053 -0.021 0.031 0.042 -0.028 0.024 
AvGrate 0.252. 0.243 0.080 0.003 0.188 0.017 0.130 
Middle School Mathematics: Cohort 1 

Ethnic School 

Model EDS EL SWD_ Female Minority Size Mean 
PP -0.438  -0.154 -0.242 -0.055  -0.262 0.135  -0.169 
MLMO -0.495 -0.065 -0.162 -0.101 -0.167 0.208 = -0.130 
Gain -0.058 0.128 0.048 -0.078 0.163 0.176 0.063 
TM -0.091 0.092 -0.031 -0.076 0.084 0.186 0.027 
SGP -0.015 0.185 0.067 -0.044 0.163 0.180 0.089 
VAM -0.026 0.170 0.053 -0.054 = 0.143 0.191 0.080 
Grate -0.123 0.063 0.010 -0.094 = 0.113 0.148 0.020 
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AvGrate 0.093 0.180 0.066 -0.006 0.130 0.109 0.095 
Middle School Mathematics: Cohort 2 
Ethnic School 

Model EDS EL SWD_ Female Minority Size Mean 
PP -0.377 = -0.266 = -0.301 0.106 = -0.255 0.081 -0.169 
MLMO -0.481 -0.197 -0.177 0.121 -0.174 0.167 = -0.124 
Gain 0.056 0.028 0.000 0.007 -0.031 0.045 0.018 
TM -0.038 -0.025 -0.081 0.022 -0.085 0.100 -0.018 
SGP 0.021 0.043 -0.048 0.025 -0.008 0.105 0.023 
VAM 0.027 0.044 -0.033 0.039 0.001 0.105 0.030 
Grate -0.026 -0.026 -0.016 0.006 -0.066 0.029  -0.016 
AvGrate 0.171 0.158 0.006 0.039 0.080 0.123 0.096 
Middle School Mathematics: Cohort 3 

Ethnic School 

Model EDS EL SWD_ Female Minority Size Mean 
PP -0.340 -0.214 -0.254 0.071 -0.121 0.140  -0.120 
MLMO -0.487 -0.187 -0.194 -0.007  -0.083 0.235 -0.120 
Gain -0.100 -0.023 -0.085 0.030 0.044 0.218 0.014 
T™ -0.152 -0.041 -0.097 0.023 0.014 0.190  -0.010 
SGP -0.050 0.026 -0.066 0.017 0.089 0.235 0.042 
VAM -0.033 0.008 -0.088 0.021 0.078 0.208 0.032 
Grate -0.189 -0.061 -0.106 0.026 0.002 0.225 -0.017 
AvGrate 0.093 0.065 -0.067 0.025 0.125 0.157 0.066 


67 


67 


Middle School Reading/Language Arts: Cohort 1 


Ethnic School 
Model EDS EL SWD_ Female Minority Size Mean 
PP -0.558 -0.350 -0.318 0.165 -0.398 0.090 = -0.228 
MLMO -0.673  -0.318 -0.289 0.085 -0.386 0.128 = -0.242 
Gain 0.327 0.166 0.113 0.002 0.184 -0.005 0.131 
TM -0.023 0.013 -0.045 0.076 -0.045 0.106 0.014 
SGP -0.059 0.033 -0.031 0.046 -0.017 0.103 0.012 
VAM -0.021 0.052 -0.034 0.010 -0.018 0.096 0.014 
Grate 0.390 0.192 0.163 -0.039 0.227 -0.010 0.154 
AvGrate 0.412 0.314 0.112 -0.038 0.242 0.044 0.181 
Middle School Reading/Language Arts: Cohort 2 
Ethnic School 
Model EDS EL SWD_ Female Minority Size Mean 
PP -0.526 -0.446 -0.221 0.204 -0.447 = -0.003  -0.240 
MLMO -0.682 -0.434 -0.185 0.188  -0.440 0.084 = -0.245 
Gain 0.395 0.259 0.133 -0.050 0.161 -0.021 0.146 
TM 0.052. 0.058 0.062 0.048 -0.013 0.053 0.043 
SGP -0.013. 0.017 0.015 0.043 -0.031 0.085 0.019 
VAM -0.028 0.014 -0.012 0.071 -0.079 0.049 0.002 
Grate 0.395 0.245 0.139  -0.081 0.139 -0.052 0.131 
AvGrate 0.469 0.332 0.148  -0.051 0.268 0.011 0.196 
Middle School Reading/Language Arts: Cohort 3 
Ethnic School 
Model EDS EL SWD_ Female Minority Size Mean 
PP -0.594 -0.360 -0.275 0.032 -0.358 0.071 = -0.247 
MLMO -0.705  -0.354 = -0.249 =0.051 -0.357 = =0.104 = -0.252 
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Gain 
TM 
SGP 
VAM 
Grate 


AvGrate 


0.292 
-0.123 
-0.153 
-0.117 
0.358 
0.313 


0.257 
0.016 
0.030 
0.037 
0.276 
0.288 


0.163 
0.034 
-0.047 
-0.009 
0.194 
0.135 


-0.014 
0.006 
0.064 
0.037 
-0.044 
0.023 


0.230 
0.053 
-0.009 
0.007 
0.255 
0.171 


0.117 
0.251 
0.168 
0.165 
0.089 
0.059 


0.174 
0.040 
0.009 
0.020 
0.188 
0.165 
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Appendix G 


Correlations of School Performance Model Estimates with School Percentage SWD for 
each Individual Cohort by Content Area and Grade Level Band. 


Elementary School Mathematics 


Cohort PP MLMO~ Gain ™ SGP VAM. Grate AvGrate 


1 -0.212 -0.185 0.066 0.004 -0.025 -0.029 0.089 -0.008 
2 -0.213 -0.198 0.015 0.003 -0.014 -0.034 0.017  -0.007 
2 -0.191 -0.169 -0.057 -0.039 -0.077 -0.097 -0.031 -0.066 


Elementary School Reading/Language Arts 


Cohort PP MLMO Gain TM SGP VAM _ Grate AvGrate 


1 -0.181 -0.178 0.120 0.076 -0.032 -0.035 0.146 0.055 
2 -0.248 -0.225 0.041 0.032 -0.079 -0.083 0.048 0.045 
3 -0.173 -0.141 0.088 0.008 -0.019 -0.028 -0.021 0.080 


Middle School Mathematics 


Cohort PP MLMO~ Gain ™ SGP VAM _ Grate AvGrate 


1 -0.242 -0.162 0.048 -0.031 0.067 0.053 0.010 0.066 
2 -0.301 -0.177 0.000 -0.081 -0.048 -0.033 -0.016 0.006 
3 -0.254 -0.194 -0.085 -0.097 -0.066 -0.088 -0.106 -0.067 


Middle School Reading/Language Arts 


Cohort PP MLMO_ Gain ™ SGP VAM _ Grate AvGrate 


1 -0.318 -0.289 0.113 -0.045 -0.031 -0.034 0.163 0.112 
2 -0.221 -0.185 0.133 0.062 0.015 -0.012 0.139 0.148 
3 -0.275 -0.249 0.163 0.034 -0.047 -0.009 0.194 0.135 
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